Jul 14 02:01:44.619073 ip-10-0-128-94 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.068811    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073869    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073881    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073885    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073889    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073892    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073895    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073897    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 02:01:45.122150 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073900    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073903    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073906    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073908    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073911    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073914    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073917    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073919    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073922    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073924    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073927    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073929    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073932    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073934    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073937    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073939    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073944    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073948    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073951    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073954    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 02:01:45.122933 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073958    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073961    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073964    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073967    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073969    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073972    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073974    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073977    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073979    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073982    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073984    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073987    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073989    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073992    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073994    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.073997    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074000    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074002    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074005    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074008    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 02:01:45.123486 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074010    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074013    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074015    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074018    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074021    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074023    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074026    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074028    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074031    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074033    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074036    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074039    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074044    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074046    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074049    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074052    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074055    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074058    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074061    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074063    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 02:01:45.124074 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074066    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074068    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074071    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074073    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074076    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074080    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074084    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074088    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074091    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074094    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074097    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074100    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074102    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074105    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074107    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074110    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074112    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074115    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074119    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 02:01:45.124667 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074121    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074124    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074126    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074129    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074132    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074135    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074138    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074141    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074143    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074146    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074149    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074151    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074154    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074880    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074887    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074890    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074893    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074896    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074899    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074902    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 02:01:45.125175 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074905    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074908    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074910    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074913    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074916    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074919    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074921    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074924    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074927    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074929    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074932    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074934    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074937    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074940    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074943    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074947    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074950    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074952    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074955    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074958    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 02:01:45.125724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074960    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074963    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074965    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074968    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074971    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074973    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074976    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074979    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074981    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074984    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074986    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074989    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074992    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074995    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.074999    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075002    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075004    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075008    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075010    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 02:01:45.126569 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075013    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075016    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075018    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075021    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075024    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075026    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075029    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075032    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075035    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075039    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075041    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075043    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075046    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075048    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075051    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075053    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075056    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075059    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075064    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075067    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 02:01:45.127152 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075069    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075071    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075075    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075079    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075082    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075085    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075087    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075090    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075093    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075095    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075098    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075101    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075104    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075106    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075109    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075111    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075114    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075117    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075119    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 02:01:45.229585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075122    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 02:01:45.139021 ip-10-0-128-94 systemd[1]: Started Kubernetes Kubelet.
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075124    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075127    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075129    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075132    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075134    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075138    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075140    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075143    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075147    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075150    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075155    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075158    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075161    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075523    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075528    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075532    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075535    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075538    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 02:01:45.230534 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075541    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075543    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075546    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075549    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075554    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075557    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075560    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075562    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075565    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075568    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075570    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075572    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075575    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075577    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075580    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075583    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075585    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075588    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075590    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075593    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 02:01:45.231628 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075595    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075598    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075601    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075603    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075607    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075609    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075612    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075615    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075617    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075621    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075623    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075626    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075628    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075631    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075633    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075636    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075640    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075643    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075646    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075648    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 02:01:45.232681 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075651    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075653    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075656    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075658    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075661    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075663    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075666    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075668    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075671    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075673    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075676    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075678    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075680    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075684    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075688    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075690    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075694    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075696    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075699    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075701    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 02:01:45.233589 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075704    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075706    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075709    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075712    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075715    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075717    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075720    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075722    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075726    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075729    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075732    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075735    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075737    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075748    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075751    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075753    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075758    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075761    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075764    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 02:01:45.234669 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075766    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075769    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075771    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075774    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075776    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075779    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075794    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075799    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075802    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075807    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075810    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075812    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075815    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075817    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.075820    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075884    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075890    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075896    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075902    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075907    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 02:01:45.235551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075911    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075920    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075926    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075930    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075933    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075937    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075940    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075943    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075947    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075949    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075952    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075955    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075958    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075962    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075964    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075968    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075971    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075974    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075978    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075981    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075984    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075989    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075992    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075995    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 02:01:45.236683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.075998    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076001    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076004    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076008    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076011    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076014    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076017    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076020    2584 flags.go:64] FLAG: --enable-server="true"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076022    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076030    2584 flags.go:64] FLAG: --event-burst="100"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076033    2584 flags.go:64] FLAG: --event-qps="50"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076036    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076039    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076042    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076052    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076055    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076058    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076061    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076064    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076067    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076070    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076073    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076076    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076078    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076081    2584 flags.go:64] FLAG: --feature-gates=""
Jul 14 02:01:45.237804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076085    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076088    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076091    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076094    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076098    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076101    2584 flags.go:64] FLAG: --help="false"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076104    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-128-94.ec2.internal"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076107    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076110    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076113    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076116    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076119    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076122    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076125    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076127    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076130    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076134    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076137    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076140    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076143    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076146    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076150    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076153    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076156    2584 flags.go:64] FLAG: --lock-file=""
Jul 14 02:01:45.239133 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076158    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076161    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076164    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076169    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076172    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076174    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076177    2584 flags.go:64] FLAG: --logging-format="text"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076180    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076183    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076186    2584 flags.go:64] FLAG: --manifest-url=""
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076188    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076192    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076197    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076201    2584 flags.go:64] FLAG: --max-pods="110"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076204    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076207    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076210    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076213    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076216    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076218    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076221    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076228    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076231    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076234    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 02:01:45.240238 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076239    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076241    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076246    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076249    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076252    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076255    2584 flags.go:64] FLAG: --port="10250"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076258    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076262    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09432dd788d42d82c"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076265    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076268    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076271    2584 flags.go:64] FLAG: --register-node="true"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076273    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076277    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076280    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076283    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076286    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076289    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076292    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076295    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076298    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076302    2584 flags.go:64] FLAG: --runonce="false"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076305    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076308    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076311    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076314    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 02:01:45.241333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076317    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076320    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076323    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076326    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076328    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076331    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076334    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076337    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076341    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076343    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076346    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076351    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076354    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076357    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076361    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076364    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076367    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076370    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076372    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076376    2584 flags.go:64] FLAG: --v="2"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076379    2584 flags.go:64] FLAG: --version="false"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076383    2584 flags.go:64] FLAG: --vmodule=""
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076386    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.076390    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076499    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 02:01:45.242451 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076502    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076505    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076510    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076512    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076515    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076518    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076521    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076524    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076527    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076529    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076532    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076535    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076537    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076540    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076542    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076546    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076550    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076553    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076555    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 02:01:45.243503 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076558    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076561    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076564    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076566    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076569    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076572    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076574    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076577    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076580    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076582    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076585    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076587    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076590    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076592    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076595    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076598    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076601    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076603    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076607    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076611    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 02:01:45.244428 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076614    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076617    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076620    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076622    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076625    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076627    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076630    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076632    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076636    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076639    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076641    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076644    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076646    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076649    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076651    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076654    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076657    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076659    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076662    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076664    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 02:01:45.245340 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076667    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076670    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076672    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076675    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076677    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076680    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076682    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076686    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076690    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076693    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076696    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076698    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076701    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076704    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076707    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076710    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076712    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076715    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076718    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 02:01:45.246257 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076720    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076725    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076727    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076730    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076733    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076737    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076740    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076743    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076746    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076749    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076751    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076754    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076757    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076759    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076762    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076764    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076767    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076769    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076772    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 02:01:45.247135 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.076774    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.077726    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.084488    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.084504    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.085175    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.085187    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085242    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085249    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085253    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085257    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085260    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085263    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085266    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085268    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 02:01:45.248373 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085272    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085275    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085277    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085280    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085283    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085286    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085289    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085291    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085294    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085296    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085299    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085302    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085304    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085307    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085309    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085312    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085315    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085317    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085320    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 02:01:45.249035 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085322    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085324    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085331    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085333    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085336    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085338    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085341    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085344    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085347    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085349    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085352    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085354    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085357    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085360    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085363    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085365    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085368    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085371    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085373    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 02:01:45.252348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085376    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085378    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085381    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085384    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085386    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085388    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085391    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085394    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085398    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085401    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085403    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085406    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085409    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085411    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085414    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085417    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085420    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085423    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085425    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085428    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 02:01:45.253256 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085430    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085433    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085435    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085438    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085440    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085444    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085447    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085450    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085452    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085455    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085458    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085461    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085463    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085466    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085468    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085471    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085474    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085478    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085481    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 02:01:46.912585 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085483    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085486    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085489    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085491    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085494    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085497    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085499    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085501    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085509    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085512    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085515    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085518    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085521    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085523    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.085528    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 02:01:46.930158 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085637    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085642    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085645    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085648    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085652    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085655    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085658    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085660    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085663    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085665    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085668    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085670    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085673    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085676    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085678    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085681    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085683    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085686    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085688    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085691    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 02:01:46.930706 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085693    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085696    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085698    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085701    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085704    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085707    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085710    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085713    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085716    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085718    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085721    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085723    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085726    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085728    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085731    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085734    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085738    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085740    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085743    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085745    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 02:01:46.931350 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085748    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085751    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085753    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085756    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085758    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085761    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085764    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085767    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085769    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085772    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085776    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085779    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085798    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085801    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085803    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085806    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085809    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085813    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085816    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 02:01:46.931922 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085818    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085822    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085826    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085828    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085831    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085833    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085837    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085840    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085843    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085848    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085851    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085854    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085856    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085859    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085862    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085864    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085867    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085870    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085872    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 02:01:46.932528 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085875    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085878    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085880    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085883    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085885    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085888    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085890    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085893    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085895    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085898    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085901    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085905    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085908    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085910    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085913    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085916    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085919    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085921    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085924    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085926    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 02:01:46.933045 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:45.085929    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.085934    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.086039    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.087894    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.089065    2584 server.go:1019] "Starting client certificate rotation"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.089167    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.089208    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.091503    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.095577    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.106659    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.110266    2584 log.go:25] "Validated CRI v1 image API"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.110922    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 02:01:46.933662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.113284    2584 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e701ab73-b7a6-4564-900b-fe1b52d44e46:/dev/nvme0n1p3 fd9f55f7-7c30-4e67-9e64-f64d8b5af6c2:/dev/nvme0n1p4]
Jul 14 02:01:46.934015 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.113299    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.118469    2584 manager.go:216] Machine: {Timestamp:2026-07-14 02:01:45.117072447 +0000 UTC m=+0.341293125 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102255 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20fd44e5f36606cc03fbe52370374f SystemUUID:ec20fd44-e5f3-6606-cc03-fbe52370374f BootID:8c582528-14ff-4ad9-b9f8-31335abc01f4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f0:04:d8:6c:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f0:04:d8:6c:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:7c:2d:ad:db:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.118578    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.118659    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.121469    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.121498    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.121677    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 02:01:46.934148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.122293    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.122306    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.122313    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.123445    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.124180    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.126826    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.126940    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.126965    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.126975    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.126988    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.126997    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.127006    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.127366    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.127386    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.129431    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.130765    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.130811    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131529    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131543    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131549    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131554    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131563    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131568    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131576    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131581    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131587    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131592    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131605    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.131612    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.132410    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.132417    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.138102    2584 server.go:1268] "Started kubelet"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.138745    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.138781    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.138915    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.140162    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.140484    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.140503    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.140968    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btmmk"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.141941    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 02:01:46.934427 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.143396    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.144409    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.144629    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.144642    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.144750    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145337    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145340    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145358    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145466    2584 factory.go:55] Registering systemd factory
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145519    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145471    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145556    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145717    2584 factory.go:153] Registering CRI-O factory
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145728    2584 factory.go:223] Registration of the crio container factory successfully
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145775    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145813    2584 factory.go:103] Registering Raw factory
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.145827    2584 manager.go:1196] Started watching for new ooms in manager
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.146389    2584 manager.go:349] Starting recovery of all containers
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.149664    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btmmk"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.157820    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.158314    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.157866    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-94.ec2.internal.18c2058ce800a1d1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-94.ec2.internal,UID:ip-10-0-128-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-94.ec2.internal,},FirstTimestamp:2026-07-14 02:01:45.138053585 +0000 UTC m=+0.362274290,LastTimestamp:2026-07-14 02:01:45.138053585 +0000 UTC m=+0.362274290,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-94.ec2.internal,}"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.166159    2584 manager.go:354] Recovery completed
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.176223    2584 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.179771    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.186772    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.186816    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.186827    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.187354    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.187364    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.187383    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 02:01:46.935671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.187356    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.188265    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.188291    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.188309    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.188863    2584 policy_none.go:49] "None policy: Start"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.188879    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.188838    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-94.ec2.internal.18c2058ceae87269  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-94.ec2.internal,UID:ip-10-0-128-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-94.ec2.internal,},FirstTimestamp:2026-07-14 02:01:45.186800233 +0000 UTC m=+0.411020915,LastTimestamp:2026-07-14 02:01:45.186800233 +0000 UTC m=+0.411020915,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-94.ec2.internal,}"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.188904    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.190274    2584 policy_none.go:47] "Start"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.280605    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282387    2584 manager.go:337] "Starting Device Plugin manager"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.282414    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282423    2584 server.go:87] "Starting device plugin registration server"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282648    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282658    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282746    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282843    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:45.282852    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.283406    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:45.283461    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.884194    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.885331    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.885366    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.885389    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:46.885426    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.887108    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:46.913981    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-94.ec2.internal\" not found" node="ip-10-0-128-94.ec2.internal"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.934115    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.935152    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.935181    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.935192    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:46.936526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.935214    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.942626    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:46.942642    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-94.ec2.internal\": node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:46.949647    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.985639    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"]
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.985695    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.987061    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.987122    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.987563    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.990305    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.990429    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.990477    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.991007    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.991018    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.991028    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.991043    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.991057    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.991043    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.992461    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.992483    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.993072    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.993098    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:46.993107    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeHasSufficientPID"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.027748    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-94.ec2.internal\" not found" node="ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.032205    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-94.ec2.internal\" not found" node="ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.073632 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.050201    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.183365 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.098595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efbaaf541750e1cbbe62f0277567641b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-94.ec2.internal\" (UID: \"efbaaf541750e1cbbe62f0277567641b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.183365 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.098621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a0d82359555acc4d2989813ad38b2615-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"a0d82359555acc4d2989813ad38b2615\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.183365 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.098640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d82359555acc4d2989813ad38b2615-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"a0d82359555acc4d2989813ad38b2615\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.183365 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.150714    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.199213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a0d82359555acc4d2989813ad38b2615-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"a0d82359555acc4d2989813ad38b2615\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.199241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d82359555acc4d2989813ad38b2615-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"a0d82359555acc4d2989813ad38b2615\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.199260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efbaaf541750e1cbbe62f0277567641b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-94.ec2.internal\" (UID: \"efbaaf541750e1cbbe62f0277567641b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.199299    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efbaaf541750e1cbbe62f0277567641b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-94.ec2.internal\" (UID: \"efbaaf541750e1cbbe62f0277567641b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.199322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a0d82359555acc4d2989813ad38b2615-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"a0d82359555acc4d2989813ad38b2615\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.199323    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d82359555acc4d2989813ad38b2615-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal\" (UID: \"a0d82359555acc4d2989813ad38b2615\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.251329    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.334425 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.329893    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.335891 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.334880    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.451482 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.352418    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.453403 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.452891    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-94.ec2.internal\" not found"
Jul 14 02:01:47.559286 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.499267    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.595795    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.596756    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.606336    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.608090    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.615702    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.639831    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.639934    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.639971    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.639976    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.678684    2584 apiserver.go:52] "Watching apiserver"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.693351    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.695351    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.697074    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.698003    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mfct5","kube-system/konnectivity-agent-r8sdt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal","openshift-multus/multus-additional-cni-plugins-rzbxv","openshift-multus/network-metrics-daemon-nmhpt","openshift-network-diagnostics/network-check-target-grzbf","openshift-network-operator/iptables-alerter-bpzhb","kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9rmq8","openshift-dns/node-resolver-lddfv","openshift-image-registry/node-ca-qkztz","openshift-multus/multus-n8vwr","openshift-ovn-kubernetes/ovnkube-node-pfrcp"]
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.700570    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.701653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.701016    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.703601 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.702226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.703601 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.703359    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.703601 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.703484    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f6p4h\""
Jul 14 02:01:47.703601 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.703595    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 02:01:47.705906 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.703830    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 02:01:47.705906 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.703891    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.705906 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.704030    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 02:01:47.709311 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.707748    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.709311 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.708078    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.709311 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.708196    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.709311 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.709079    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.709311 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.709239    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.709320    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.709473    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f47w5\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.709877    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.709966    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.710197    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkwgl\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.710298    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fhrl\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.710505    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 02:01:47.711391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.711022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 02:01:47.713733 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.712455    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:47.713733 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.712697    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:47.714316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.713904    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 02:01:47.714316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.713916    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.714316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.713930    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 02:01:47.714316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.713945    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.714364    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 01:56:46 +0000 UTC" deadline="2028-01-14 14:53:59.328475453 +0000 UTC"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.714414    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13188h52m11.61406588s"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.714696    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jmbj7\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.714763    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.714806    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.715002    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.715977    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.716018    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.716244    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.717199    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.718384    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.719268    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.719339    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74h9l\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.719535    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.719538    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.719659    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.720649    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.720656    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.720672    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.720852    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5r6pp\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.720898    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.721650    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d26rw\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.721655    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.722961    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.723064    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bb4ff\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.733490    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.733938    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8xxfk"
Jul 14 02:01:47.786404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.737996    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8xxfk"
Jul 14 02:01:47.799486 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.797538    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.803977    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-kubernetes\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.803998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-lib-modules\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-etc-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-system-cni-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804053    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-cni-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-os-release\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysctl-conf\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj558\" (UniqueName: \"kubernetes.io/projected/53f09255-d824-4fed-b5a4-bb594e253ada-kube-api-access-hj558\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804175    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adce1978-62d1-4533-a4b9-b8a4703092aa-kubelet-config\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804209    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec5e2f93-6554-4cab-9107-a4e693952e62-tmp\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkql\" (UniqueName: \"kubernetes.io/projected/ec5e2f93-6554-4cab-9107-a4e693952e62-kube-api-access-xgkql\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-var-lib-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804270    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-env-overrides\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.892029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804300    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adce1978-62d1-4533-a4b9-b8a4703092aa-dbus\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-cnibin\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804343    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-multus-certs\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8eda87a6-bdd6-4ca3-9213-43e673eec75b-konnectivity-ca\") pod \"konnectivity-agent-r8sdt\" (UID: \"8eda87a6-bdd6-4ca3-9213-43e673eec75b\") " pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804378    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804405    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-run-netns\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804457    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovnkube-config\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a70b3316-2d49-4928-9428-88d957426bf7-host\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-socket-dir-parent\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-netns\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804547    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-os-release\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.893066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804570    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-registration-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-hostroot\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-cnibin\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804644    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-cni-netd\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-cni-multus\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81f850e6-9a94-4e26-9880-39802130d134-tmp-dir\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-modprobe-d\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-var-lib-kubelet\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krhl\" (UniqueName: \"kubernetes.io/projected/1dc1b087-17d0-42b5-affa-a6eff7db68f4-kube-api-access-9krhl\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-cni-bin\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804832    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckrlb\" (UniqueName: \"kubernetes.io/projected/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-kube-api-access-ckrlb\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.894795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysctl-d\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804858    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-system-cni-dir\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-systemd\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-host-slash\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.804911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81f850e6-9a94-4e26-9880-39802130d134-hosts-file\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a067b638-6577-411c-a021-a92848fe5508-cni-binary-copy\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-iptables-alerter-script\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805240    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-run\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805266    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-tuned\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-socket-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805333    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-ovn\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.895615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-systemd\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805376    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8thk\" (UniqueName: \"kubernetes.io/projected/a296f1cc-563f-4feb-96fe-c4c56ef944b9-kube-api-access-w8thk\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-systemd-units\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805430    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-node-log\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-k8s-cni-cncf-io\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-kubelet\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-host\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-log-socket\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805537    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-conf-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-etc-kubernetes\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnjt\" (UniqueName: \"kubernetes.io/projected/a067b638-6577-411c-a021-a92848fe5508-kube-api-access-bjnjt\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805608    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klmz\" (UniqueName: \"kubernetes.io/projected/81f850e6-9a94-4e26-9880-39802130d134-kube-api-access-5klmz\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:47.966272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805642    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovn-node-metrics-cert\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-sys\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-sys-fs\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-kubelet\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-cni-bin\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a067b638-6577-411c-a021-a92848fe5508-multus-daemon-config\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8eda87a6-bdd6-4ca3-9213-43e673eec75b-agent-certs\") pod \"konnectivity-agent-r8sdt\" (UID: \"8eda87a6-bdd6-4ca3-9213-43e673eec75b\") " pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2zp\" (UniqueName: \"kubernetes.io/projected/02cc23f1-305c-4b98-945c-bc9887836923-kube-api-access-ph2zp\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805817    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-slash\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805830    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovnkube-script-lib\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805881    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a70b3316-2d49-4928-9428-88d957426bf7-serviceca\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-device-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805912    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84k4c\" (UniqueName: \"kubernetes.io/projected/a70b3316-2d49-4928-9428-88d957426bf7-kube-api-access-84k4c\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.967717 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.805934    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysconfig\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906243    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-cnibin\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-cnibin\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-cni-netd\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906341    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-cni-multus\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81f850e6-9a94-4e26-9880-39802130d134-tmp-dir\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-modprobe-d\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-cni-netd\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-var-lib-kubelet\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-cni-multus\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krhl\" (UniqueName: \"kubernetes.io/projected/1dc1b087-17d0-42b5-affa-a6eff7db68f4-kube-api-access-9krhl\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-var-lib-kubelet\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-cni-bin\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckrlb\" (UniqueName: \"kubernetes.io/projected/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-kube-api-access-ckrlb\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-modprobe-d\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysctl-d\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-cni-bin\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-system-cni-dir\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-systemd\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-host-slash\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81f850e6-9a94-4e26-9880-39802130d134-hosts-file\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906641    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-system-cni-dir\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906657    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-systemd\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906640    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysctl-d\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81f850e6-9a94-4e26-9880-39802130d134-tmp-dir\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906690    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81f850e6-9a94-4e26-9880-39802130d134-hosts-file\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.973926 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-host-slash\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.906748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a067b638-6577-411c-a021-a92848fe5508-cni-binary-copy\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-iptables-alerter-script\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-run\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907201    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-tuned\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-run\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907347    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907390    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-socket-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-ovn\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-systemd\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907477    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.974572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8thk\" (UniqueName: \"kubernetes.io/projected/a296f1cc-563f-4feb-96fe-c4c56ef944b9-kube-api-access-w8thk\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907560    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-ovn\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-systemd-units\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907595    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-node-log\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907647    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-socket-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-k8s-cni-cncf-io\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-kubelet\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907699    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-k8s-cni-cncf-io\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907724    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-host\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-log-socket\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-conf-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-iptables-alerter-script\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-host\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-etc-kubernetes\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.975810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnjt\" (UniqueName: \"kubernetes.io/projected/a067b638-6577-411c-a021-a92848fe5508-kube-api-access-bjnjt\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klmz\" (UniqueName: \"kubernetes.io/projected/81f850e6-9a94-4e26-9880-39802130d134-kube-api-access-5klmz\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907984    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-systemd-units\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovn-node-metrics-cert\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-sys\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-node-log\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-sys-fs\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-kubelet\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908108    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-log-socket\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908119    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-cni-bin\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a067b638-6577-411c-a021-a92848fe5508-multus-daemon-config\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8eda87a6-bdd6-4ca3-9213-43e673eec75b-agent-certs\") pod \"konnectivity-agent-r8sdt\" (UID: \"8eda87a6-bdd6-4ca3-9213-43e673eec75b\") " pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.908213    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2zp\" (UniqueName: \"kubernetes.io/projected/02cc23f1-305c-4b98-945c-bc9887836923-kube-api-access-ph2zp\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.976374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-slash\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.908287    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:48.408268441 +0000 UTC m=+2.081143243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908308    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-slash\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovnkube-script-lib\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a067b638-6577-411c-a021-a92848fe5508-cni-binary-copy\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a70b3316-2d49-4928-9428-88d957426bf7-serviceca\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-device-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908412    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-conf-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84k4c\" (UniqueName: \"kubernetes.io/projected/a70b3316-2d49-4928-9428-88d957426bf7-kube-api-access-84k4c\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysconfig\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-kubelet\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-kubernetes\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-lib-modules\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.976909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-etc-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908580    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-system-cni-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-cni-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-os-release\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysctl-conf\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj558\" (UniqueName: \"kubernetes.io/projected/53f09255-d824-4fed-b5a4-bb594e253ada-kube-api-access-hj558\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adce1978-62d1-4533-a4b9-b8a4703092aa-kubelet-config\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908757    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec5e2f93-6554-4cab-9107-a4e693952e62-tmp\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908785    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkql\" (UniqueName: \"kubernetes.io/projected/ec5e2f93-6554-4cab-9107-a4e693952e62-kube-api-access-xgkql\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-var-lib-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908854    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a067b638-6577-411c-a021-a92848fe5508-multus-daemon-config\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908876    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-env-overrides\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adce1978-62d1-4533-a4b9-b8a4703092aa-dbus\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908944    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-cnibin\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.908974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-multus-certs\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8eda87a6-bdd6-4ca3-9213-43e673eec75b-konnectivity-ca\") pod \"konnectivity-agent-r8sdt\" (UID: \"8eda87a6-bdd6-4ca3-9213-43e673eec75b\") " pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-run-netns\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909177    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovnkube-config\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a70b3316-2d49-4928-9428-88d957426bf7-host\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909272    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-cni-bin\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-env-overrides\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-kubernetes\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909386    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adce1978-62d1-4533-a4b9-b8a4703092aa-dbus\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.907512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-systemd\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909439    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-cnibin\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-lib-modules\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-etc-kubernetes\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.977907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909505    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-run-netns\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909582    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-multus-certs\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.909948    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysctl-conf\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-etc-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-system-cni-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910378    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovnkube-script-lib\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910413    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-os-release\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8eda87a6-bdd6-4ca3-9213-43e673eec75b-konnectivity-ca\") pod \"konnectivity-agent-r8sdt\" (UID: \"8eda87a6-bdd6-4ca3-9213-43e673eec75b\") " pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-var-lib-kubelet\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-sys\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-socket-dir-parent\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910574    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a70b3316-2d49-4928-9428-88d957426bf7-host\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910574    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adce1978-62d1-4533-a4b9-b8a4703092aa-kubelet-config\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.978551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-run-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910904    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-netns\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-os-release\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.910967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-registration-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-tuned\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-hostroot\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-host-run-netns\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1dc1b087-17d0-42b5-affa-a6eff7db68f4-var-lib-openvswitch\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-device-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-sys-fs\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-socket-dir-parent\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.979045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911296    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec5e2f93-6554-4cab-9107-a4e693952e62-etc-sysconfig\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911345    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a70b3316-2d49-4928-9428-88d957426bf7-serviceca\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.911429    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-hostroot\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/53f09255-d824-4fed-b5a4-bb594e253ada-os-release\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a067b638-6577-411c-a021-a92848fe5508-multus-cni-dir\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.911614    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:01:48.411584029 +0000 UTC m=+2.084458823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911643    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovnkube-config\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.911944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f09255-d824-4fed-b5a4-bb594e253ada-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.912500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02cc23f1-305c-4b98-945c-bc9887836923-registration-dir\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.912995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1dc1b087-17d0-42b5-affa-a6eff7db68f4-ovn-node-metrics-cert\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.913189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8eda87a6-bdd6-4ca3-9213-43e673eec75b-agent-certs\") pod \"konnectivity-agent-r8sdt\" (UID: \"8eda87a6-bdd6-4ca3-9213-43e673eec75b\") " pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.913438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec5e2f93-6554-4cab-9107-a4e693952e62-tmp\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.914682    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.914706    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:01:47.981363 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.914720    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:47.914790    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:48.414774877 +0000 UTC m=+2.087649670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.915169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krhl\" (UniqueName: \"kubernetes.io/projected/1dc1b087-17d0-42b5-affa-a6eff7db68f4-kube-api-access-9krhl\") pod \"ovnkube-node-pfrcp\" (UID: \"1dc1b087-17d0-42b5-affa-a6eff7db68f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.915787    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8thk\" (UniqueName: \"kubernetes.io/projected/a296f1cc-563f-4feb-96fe-c4c56ef944b9-kube-api-access-w8thk\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.917512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2zp\" (UniqueName: \"kubernetes.io/projected/02cc23f1-305c-4b98-945c-bc9887836923-kube-api-access-ph2zp\") pod \"aws-ebs-csi-driver-node-zzdvk\" (UID: \"02cc23f1-305c-4b98-945c-bc9887836923\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.921262    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84k4c\" (UniqueName: \"kubernetes.io/projected/a70b3316-2d49-4928-9428-88d957426bf7-kube-api-access-84k4c\") pod \"node-ca-qkztz\" (UID: \"a70b3316-2d49-4928-9428-88d957426bf7\") " pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.921624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klmz\" (UniqueName: \"kubernetes.io/projected/81f850e6-9a94-4e26-9880-39802130d134-kube-api-access-5klmz\") pod \"node-resolver-lddfv\" (UID: \"81f850e6-9a94-4e26-9880-39802130d134\") " pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.921885    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj558\" (UniqueName: \"kubernetes.io/projected/53f09255-d824-4fed-b5a4-bb594e253ada-kube-api-access-hj558\") pod \"multus-additional-cni-plugins-rzbxv\" (UID: \"53f09255-d824-4fed-b5a4-bb594e253ada\") " pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.921921    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkql\" (UniqueName: \"kubernetes.io/projected/ec5e2f93-6554-4cab-9107-a4e693952e62-kube-api-access-xgkql\") pod \"tuned-9rmq8\" (UID: \"ec5e2f93-6554-4cab-9107-a4e693952e62\") " pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.942154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.953301    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnjt\" (UniqueName: \"kubernetes.io/projected/a067b638-6577-411c-a021-a92848fe5508-kube-api-access-bjnjt\") pod \"multus-n8vwr\" (UID: \"a067b638-6577-411c-a021-a92848fe5508\") " pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:47.981780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:47.954043    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckrlb\" (UniqueName: \"kubernetes.io/projected/0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d-kube-api-access-ckrlb\") pod \"iptables-alerter-bpzhb\" (UID: \"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d\") " pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:48.017652 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.017618    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d82359555acc4d2989813ad38b2615.slice/crio-62597a68e5d849adb74001c5988f9e6da424d4a1368ae39426e9468934ae32c8 WatchSource:0}: Error finding container 62597a68e5d849adb74001c5988f9e6da424d4a1368ae39426e9468934ae32c8: Status 404 returned error can't find the container with id 62597a68e5d849adb74001c5988f9e6da424d4a1368ae39426e9468934ae32c8
Jul 14 02:01:48.018097 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.018078    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbaaf541750e1cbbe62f0277567641b.slice/crio-0cac5053adefc484f6e85118f6ea30fb1f8ca14f4501784a222d2da647224156 WatchSource:0}: Error finding container 0cac5053adefc484f6e85118f6ea30fb1f8ca14f4501784a222d2da647224156: Status 404 returned error can't find the container with id 0cac5053adefc484f6e85118f6ea30fb1f8ca14f4501784a222d2da647224156
Jul 14 02:01:48.022155 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.022124    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 02:01:48.042278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.042261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:01:48.049505 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.049485    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc1b087_17d0_42b5_affa_a6eff7db68f4.slice/crio-639039d20ce2f1e30fc6fdb6d4a04f2bc40387283bdcd242aa6a456dc8612aa3 WatchSource:0}: Error finding container 639039d20ce2f1e30fc6fdb6d4a04f2bc40387283bdcd242aa6a456dc8612aa3: Status 404 returned error can't find the container with id 639039d20ce2f1e30fc6fdb6d4a04f2bc40387283bdcd242aa6a456dc8612aa3
Jul 14 02:01:48.052448 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.052432    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bpzhb"
Jul 14 02:01:48.080173 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.080156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lddfv"
Jul 14 02:01:48.086262 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.086242    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f850e6_9a94_4e26_9880_39802130d134.slice/crio-b28645393135cb26ac6d7a519b74e104972186c16b6b2fecac843bfef1b86a3f WatchSource:0}: Error finding container b28645393135cb26ac6d7a519b74e104972186c16b6b2fecac843bfef1b86a3f: Status 404 returned error can't find the container with id b28645393135cb26ac6d7a519b74e104972186c16b6b2fecac843bfef1b86a3f
Jul 14 02:01:48.137904 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.137876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9rmq8"
Jul 14 02:01:48.141680 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.141665    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzbxv"
Jul 14 02:01:48.144522 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.144501    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5e2f93_6554_4cab_9107_a4e693952e62.slice/crio-32f43aa31c3d5425d812f311ea874d1aaef037db6697e31b414405767330226b WatchSource:0}: Error finding container 32f43aa31c3d5425d812f311ea874d1aaef037db6697e31b414405767330226b: Status 404 returned error can't find the container with id 32f43aa31c3d5425d812f311ea874d1aaef037db6697e31b414405767330226b
Jul 14 02:01:48.148866 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.148844    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f09255_d824_4fed_b5a4_bb594e253ada.slice/crio-917929b681beb353b682ef52622cdd7b22fa236bf3c9a6246c5dcdf505e539c1 WatchSource:0}: Error finding container 917929b681beb353b682ef52622cdd7b22fa236bf3c9a6246c5dcdf505e539c1: Status 404 returned error can't find the container with id 917929b681beb353b682ef52622cdd7b22fa236bf3c9a6246c5dcdf505e539c1
Jul 14 02:01:48.164690 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.164669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk"
Jul 14 02:01:48.171244 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.171221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cc23f1_305c_4b98_945c_bc9887836923.slice/crio-6788593dd2f6003fd0218f093e6e3d6e3c7393f855c6cedcdd178da8bdf15252 WatchSource:0}: Error finding container 6788593dd2f6003fd0218f093e6e3d6e3c7393f855c6cedcdd178da8bdf15252: Status 404 returned error can't find the container with id 6788593dd2f6003fd0218f093e6e3d6e3c7393f855c6cedcdd178da8bdf15252
Jul 14 02:01:48.186057 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.186036    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qkztz"
Jul 14 02:01:48.192477 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.192458    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70b3316_2d49_4928_9428_88d957426bf7.slice/crio-5e481aa06107de23b9eb2d6f7a4c16c071e860a16fa3fbc621ff427aff49cfc2 WatchSource:0}: Error finding container 5e481aa06107de23b9eb2d6f7a4c16c071e860a16fa3fbc621ff427aff49cfc2: Status 404 returned error can't find the container with id 5e481aa06107de23b9eb2d6f7a4c16c071e860a16fa3fbc621ff427aff49cfc2
Jul 14 02:01:48.236898 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.236844    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n8vwr"
Jul 14 02:01:48.243080 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.243058    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda067b638_6577_411c_a021_a92848fe5508.slice/crio-752f96d1e0392c4ba0caa3dad40ddeddd4e74b095b6cc1b9c8fb987e025e7527 WatchSource:0}: Error finding container 752f96d1e0392c4ba0caa3dad40ddeddd4e74b095b6cc1b9c8fb987e025e7527: Status 404 returned error can't find the container with id 752f96d1e0392c4ba0caa3dad40ddeddd4e74b095b6cc1b9c8fb987e025e7527
Jul 14 02:01:48.299504 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:01:48.299483    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eda87a6_bdd6_4ca3_9213_43e673eec75b.slice/crio-66765bc6136f624a8294d1c1fd4ce828ca1ab24646459d6695cfb0f638610326 WatchSource:0}: Error finding container 66765bc6136f624a8294d1c1fd4ce828ca1ab24646459d6695cfb0f638610326: Status 404 returned error can't find the container with id 66765bc6136f624a8294d1c1fd4ce828ca1ab24646459d6695cfb0f638610326
Jul 14 02:01:48.415344 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.415304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:48.415517 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.415356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:48.415517 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.415402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:48.415517 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415473    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:01:48.415517 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415490    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:48.415517 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415497    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:01:48.415517 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415516    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:48.415719 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415490    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:48.415719 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415549    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:49.415530812 +0000 UTC m=+3.088405592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:48.415719 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415566    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:49.415557129 +0000 UTC m=+3.088431915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:48.415719 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.415579    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:01:49.415572298 +0000 UTC m=+3.088447077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:48.740237 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.740176    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 01:56:47 +0000 UTC" deadline="2028-04-24 06:15:11.274353366 +0000 UTC"
Jul 14 02:01:48.740237 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.740237    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15604h13m22.534119957s"
Jul 14 02:01:48.794068 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.794039    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:48.887278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.887252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:48.887447 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.887384    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:48.888040 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.887875    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:48.888040 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:48.887971    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:48.920514 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.920104    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal" event={"ID":"efbaaf541750e1cbbe62f0277567641b","Type":"ContainerStarted","Data":"0cac5053adefc484f6e85118f6ea30fb1f8ca14f4501784a222d2da647224156"}
Jul 14 02:01:48.925807 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.925779    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qkztz" event={"ID":"a70b3316-2d49-4928-9428-88d957426bf7","Type":"ContainerStarted","Data":"5e481aa06107de23b9eb2d6f7a4c16c071e860a16fa3fbc621ff427aff49cfc2"}
Jul 14 02:01:48.930720 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.930691    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk" event={"ID":"02cc23f1-305c-4b98-945c-bc9887836923","Type":"ContainerStarted","Data":"6788593dd2f6003fd0218f093e6e3d6e3c7393f855c6cedcdd178da8bdf15252"}
Jul 14 02:01:48.945228 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.945201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerStarted","Data":"917929b681beb353b682ef52622cdd7b22fa236bf3c9a6246c5dcdf505e539c1"}
Jul 14 02:01:48.956944 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.956915    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" event={"ID":"a0d82359555acc4d2989813ad38b2615","Type":"ContainerStarted","Data":"62597a68e5d849adb74001c5988f9e6da424d4a1368ae39426e9468934ae32c8"}
Jul 14 02:01:48.972173 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.972144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r8sdt" event={"ID":"8eda87a6-bdd6-4ca3-9213-43e673eec75b","Type":"ContainerStarted","Data":"66765bc6136f624a8294d1c1fd4ce828ca1ab24646459d6695cfb0f638610326"}
Jul 14 02:01:48.987175 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:48.987147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n8vwr" event={"ID":"a067b638-6577-411c-a021-a92848fe5508","Type":"ContainerStarted","Data":"752f96d1e0392c4ba0caa3dad40ddeddd4e74b095b6cc1b9c8fb987e025e7527"}
Jul 14 02:01:49.010821 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.010753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9rmq8" event={"ID":"ec5e2f93-6554-4cab-9107-a4e693952e62","Type":"ContainerStarted","Data":"32f43aa31c3d5425d812f311ea874d1aaef037db6697e31b414405767330226b"}
Jul 14 02:01:49.031754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.031673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lddfv" event={"ID":"81f850e6-9a94-4e26-9880-39802130d134","Type":"ContainerStarted","Data":"b28645393135cb26ac6d7a519b74e104972186c16b6b2fecac843bfef1b86a3f"}
Jul 14 02:01:49.060639 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.060607    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bpzhb" event={"ID":"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d","Type":"ContainerStarted","Data":"383a32a9064832c3a84e72e3b5d4e198cd6f14532cc27117f694c25437847a17"}
Jul 14 02:01:49.083754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.083666    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"639039d20ce2f1e30fc6fdb6d4a04f2bc40387283bdcd242aa6a456dc8612aa3"}
Jul 14 02:01:49.424204 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.423993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:49.424204 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.424043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:49.424204 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.424105    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424238    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424247    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424275    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424289    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424297    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:01:51.424278565 +0000 UTC m=+5.097153347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424342    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:51.424323971 +0000 UTC m=+5.097198750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424374    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:49.424551 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.424424    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:51.424411912 +0000 UTC m=+5.097286705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:49.741492 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.741358    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 01:56:47 +0000 UTC" deadline="2028-04-11 12:42:52.421442068 +0000 UTC"
Jul 14 02:01:49.741492 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.741421    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15298h41m2.680025341s"
Jul 14 02:01:49.800072 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.800043    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:49.887006 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:49.886514    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:49.887006 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:49.886638    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:01:50.506278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:50.506021    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 02:01:50.889824 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:50.888247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:50.889824 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:50.888367    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:50.889824 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:50.888860    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:50.889824 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:50.888961    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:51.441150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:51.441111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:51.441338 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:51.441175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:51.441338 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:51.441198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:51.441338 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441282    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:51.441338 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441325    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:55.441313149 +0000 UTC m=+9.114187927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:51.441631 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441610    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:51.441753 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441648    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:01:55.441638431 +0000 UTC m=+9.114513209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:51.441753 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441715    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:01:51.441753 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441732    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:01:51.441753 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441743    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:51.441945 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.441766    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:01:55.441758987 +0000 UTC m=+9.114633764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:51.886234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:51.886155    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:51.886390 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:51.886295    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:01:52.888704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:52.888228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:52.888704 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:52.888346    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:52.888704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:52.888407    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:52.888704 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:52.888493    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:53.885843 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:53.885808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:53.885988 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:53.885917    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:01:54.887895 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:54.887360    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:54.887895 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:54.887484    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:54.887895 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:54.887360    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:54.888470 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:54.887924    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:55.474885 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:55.474616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:55.474885 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:55.474685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:55.474885 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:55.474716    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:55.474885 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.474849    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.474909    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:03.474890717 +0000 UTC m=+17.147765501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.474976    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.475049    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:02:03.475029934 +0000 UTC m=+17.147904712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.475098    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.475122    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.475149    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:55.475277 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.475201    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:03.475187186 +0000 UTC m=+17.148061973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:01:55.885878 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:55.885848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:55.886038 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:55.885979    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:01:56.887590 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:56.887553    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:56.888034 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:56.887676    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:56.888034 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:56.887703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:56.888034 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:56.887819    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:57.886597 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:57.886554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:57.886782 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:57.886693    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:01:58.885929 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:58.885895    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:01:58.886372 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:58.885903    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:01:58.886372 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:58.886025    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:01:58.886372 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:58.886122    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:01:59.886154 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:01:59.886105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:01:59.886546 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:01:59.886234    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:00.888708 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:00.888677    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:00.889153 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:00.888783    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:00.889153 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:00.888886    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:00.889153 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:00.888989    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:01.886066 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:01.886031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:01.886271 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:01.886169    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:02.889288 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:02.889254    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:02.889702 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:02.889354    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:02.889702 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:02.889421    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:02.889702 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:02.889516    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:03.536463 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:03.536434    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:03.536463 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:03.536470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:03.536663 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536598    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:02:03.536663 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536621    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:02:03.536663 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536631    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:02:03.536663 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:03.536632    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:03.536811 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536680    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:19.536661079 +0000 UTC m=+33.209535871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:02:03.536811 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536683    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:02:03.536811 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536620    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:02:03.536811 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536730    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:02:19.536714119 +0000 UTC m=+33.209588916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:02:03.536811 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.536762    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:19.536742095 +0000 UTC m=+33.209616873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:02:03.885949 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:03.885923    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:03.886092 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:03.886045    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:04.886411 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:04.886377    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:04.886836 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:04.886501    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:04.886836 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:04.886562    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:04.886836 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:04.886669    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:05.886568 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:05.886531    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:05.887037 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:05.886654    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:06.887498 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:06.887463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:06.887498 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:06.887496    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:06.887929 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:06.887568    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:06.887929 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:06.887790    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:07.885896 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:07.885872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:07.885983 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:07.885967    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:08.886281 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:08.885962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:08.886881 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:08.886017    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:08.886881 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:08.886369    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:08.886881 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:08.886410    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:09.142541 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.142514    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qkztz" event={"ID":"a70b3316-2d49-4928-9428-88d957426bf7","Type":"ContainerStarted","Data":"db11c7976a9a298e14002837d402f7e5d61301a7a284a926f64fac152e8bb91f"}
Jul 14 02:02:09.144094 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.144070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk" event={"ID":"02cc23f1-305c-4b98-945c-bc9887836923","Type":"ContainerStarted","Data":"251150e287944de50ef1424b2435ed617564dfa7afedec47ccf37acdffba2321"}
Jul 14 02:02:09.145508 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.145486    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f09255-d824-4fed-b5a4-bb594e253ada" containerID="bba5d893f6a45a9d46fbf7f07e78f772e0fc16f600d77dfd6333f39956f84584" exitCode=0
Jul 14 02:02:09.145599 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.145550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerDied","Data":"bba5d893f6a45a9d46fbf7f07e78f772e0fc16f600d77dfd6333f39956f84584"}
Jul 14 02:02:09.147150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.147109    2584 generic.go:356] "Generic (PLEG): container finished" podID="a0d82359555acc4d2989813ad38b2615" containerID="4156c320329c16cdd8fb7d004bcca47569913f11bb66b9e3c5bc29dfb2ce358d" exitCode=0
Jul 14 02:02:09.147266 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.147165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" event={"ID":"a0d82359555acc4d2989813ad38b2615","Type":"ContainerDied","Data":"4156c320329c16cdd8fb7d004bcca47569913f11bb66b9e3c5bc29dfb2ce358d"}
Jul 14 02:02:09.148560 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.148540    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r8sdt" event={"ID":"8eda87a6-bdd6-4ca3-9213-43e673eec75b","Type":"ContainerStarted","Data":"ac6b6e38e734a95b8edc69d8da94dd4edee823641cdb2df2793d672d39b752a9"}
Jul 14 02:02:09.149960 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.149940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n8vwr" event={"ID":"a067b638-6577-411c-a021-a92848fe5508","Type":"ContainerStarted","Data":"a51ddecdab4191a288785094624e151a31d5b2701bd638cba61d50d002bf9901"}
Jul 14 02:02:09.151416 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.151397    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9rmq8" event={"ID":"ec5e2f93-6554-4cab-9107-a4e693952e62","Type":"ContainerStarted","Data":"8ecc470c09a273a904b4210146705f52577dcfd371ffbbc4256018a85d9349ec"}
Jul 14 02:02:09.152751 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.152721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lddfv" event={"ID":"81f850e6-9a94-4e26-9880-39802130d134","Type":"ContainerStarted","Data":"96fe3ebd1c5113fd789e0d7ee111852e6f6847f380c9800f8a2582bcfe893c63"}
Jul 14 02:02:09.155160 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.155124    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"10d9848e2c410c265a212feed1e4729820c211d8e508234f5dd81fac2bb937c4"}
Jul 14 02:02:09.155239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.155196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"4064235aa5f22de7e519a8e2433dd6072cbe45f0ccb84b2c4d1bb14e2ec16d01"}
Jul 14 02:02:09.155239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.155230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"68f02fb02d89e27f3b68c9624412f38dc3e3b65cadf7107ddeacf30d9d35d81d"}
Jul 14 02:02:09.155319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.155258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"1d1058f29fc3968c59dc46c34d4d3167f689c5fe51bb34492aca12af607b87cb"}
Jul 14 02:02:09.155319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.155297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"22a71d87800306d554cd0426880b6b9f5717ff37f72cde3d61d0e94a930a9c8b"}
Jul 14 02:02:09.156511 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.156494    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal" event={"ID":"efbaaf541750e1cbbe62f0277567641b","Type":"ContainerStarted","Data":"b64cdd8c0acb8f7a03a5cd1488308b9a660e9ed2ddf8f82f34b86a15eaa5ebde"}
Jul 14 02:02:09.159564 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.159505    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qkztz" podStartSLOduration=2.281721762 podStartE2EDuration="22.159489841s" totalImagesPullingTime="19.877768079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:09.158720571 +0000 UTC m=+22.831595372" watchObservedRunningTime="2026-07-14 02:02:09.159489841 +0000 UTC m=+22.832364642"
Jul 14 02:02:09.173095 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.173061    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r8sdt" podStartSLOduration=2.767653975 podStartE2EDuration="22.173048793s" totalImagesPullingTime="19.405394818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:09.172556927 +0000 UTC m=+22.845431728" watchObservedRunningTime="2026-07-14 02:02:09.173048793 +0000 UTC m=+22.845923606"
Jul 14 02:02:09.207300 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.207248    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n8vwr" podStartSLOduration=2.180168931 podStartE2EDuration="22.207229594s" totalImagesPullingTime="20.027060663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:09.207081692 +0000 UTC m=+22.879956474" watchObservedRunningTime="2026-07-14 02:02:09.207229594 +0000 UTC m=+22.880104395"
Jul 14 02:02:09.221150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.221069    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lddfv" podStartSLOduration=3.23582299 podStartE2EDuration="23.221049925s" totalImagesPullingTime="19.985226935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:09.220836189 +0000 UTC m=+22.893710989" watchObservedRunningTime="2026-07-14 02:02:09.221049925 +0000 UTC m=+22.893924727"
Jul 14 02:02:09.240493 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.240449    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9rmq8" podStartSLOduration=3.313234218 podStartE2EDuration="23.240435912s" totalImagesPullingTime="19.927201694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:09.239890454 +0000 UTC m=+22.912765255" watchObservedRunningTime="2026-07-14 02:02:09.240435912 +0000 UTC m=+22.913310711"
Jul 14 02:02:09.287396 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.287342    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-94.ec2.internal" podStartSLOduration=22.287322712 podStartE2EDuration="22.287322712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:09.286668096 +0000 UTC m=+22.959542897" watchObservedRunningTime="2026-07-14 02:02:09.287322712 +0000 UTC m=+22.960197515"
Jul 14 02:02:09.646176 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.646154    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 02:02:09.847596 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.847469    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T02:02:09.646170199Z","UUID":"a93a3594-cbc0-4b7a-b92e-be5b2ebd11d7","Handler":null,"Name":"","Endpoint":""}
Jul 14 02:02:09.849188 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.849166    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 02:02:09.849329 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.849205    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 02:02:09.886598 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:09.886576    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:09.887015 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:09.886711    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:10.160842 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.160755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bpzhb" event={"ID":"0d7ad732-ebdf-4aa9-83fb-e5a6a356a02d","Type":"ContainerStarted","Data":"1ed51261ae60cb4be39734a528d975fb8f40ef6fa6c202d9b60ca0ebc94b11c0"}
Jul 14 02:02:10.164661 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.164634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"89898c3a70aba86c9939da86fb1c34e5bea3fd8d162fa46d03d3497a5ea5a707"}
Jul 14 02:02:10.167196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.167165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk" event={"ID":"02cc23f1-305c-4b98-945c-bc9887836923","Type":"ContainerStarted","Data":"1e15755e85bec4a50b34595a9ca18071335c05f7581707e6e6ce73c510f0acc4"}
Jul 14 02:02:10.169348 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.169321    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" event={"ID":"a0d82359555acc4d2989813ad38b2615","Type":"ContainerStarted","Data":"0bff98b7781ad018d55ef67828e0c7b8fbc34baf56407a697f8120130fc7c704"}
Jul 14 02:02:10.176147 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.176097    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bpzhb" podStartSLOduration=4.53589341 podStartE2EDuration="24.176079526s" totalImagesPullingTime="19.640186116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:10.175175545 +0000 UTC m=+23.848050347" watchObservedRunningTime="2026-07-14 02:02:10.176079526 +0000 UTC m=+23.848954347"
Jul 14 02:02:10.189438 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.189391    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-94.ec2.internal" podStartSLOduration=23.18937485 podStartE2EDuration="23.18937485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:10.189198682 +0000 UTC m=+23.862073482" watchObservedRunningTime="2026-07-14 02:02:10.18937485 +0000 UTC m=+23.862249653"
Jul 14 02:02:10.886702 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.886669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:10.887047 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:10.886819    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:10.887208 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:10.887188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:10.887297 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:10.887275    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:11.173978 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:11.173935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk" event={"ID":"02cc23f1-305c-4b98-945c-bc9887836923","Type":"ContainerStarted","Data":"2f3ad30441fce3b157dc3ac33b2ddb79bd518f73aabb2bfc29c7832d8fa14066"}
Jul 14 02:02:11.192052 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:11.191999    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzdvk" podStartSLOduration=2.521648129 podStartE2EDuration="24.191984777s" totalImagesPullingTime="21.670336648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:11.191430726 +0000 UTC m=+24.864305527" watchObservedRunningTime="2026-07-14 02:02:11.191984777 +0000 UTC m=+24.864859579"
Jul 14 02:02:11.613699 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:11.613626    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:02:11.614732 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:11.614711    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:02:11.885903 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:11.885871    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:11.886194 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:11.886005    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:12.180245 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:12.180173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"3e26cb918524a3ef3aacd521b09045722c69c228cfec8b288f3b27671e81a3d2"}
Jul 14 02:02:12.180770 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:12.180484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:02:12.181113 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:12.181097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r8sdt"
Jul 14 02:02:12.886308 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:12.886055    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:12.886488 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:12.886123    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:12.886488 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:12.886427    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:12.886488 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:12.886455    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:13.886025 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:13.885856    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:13.886723 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:13.886151    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:14.188208 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.188069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" event={"ID":"1dc1b087-17d0-42b5-affa-a6eff7db68f4","Type":"ContainerStarted","Data":"fcd538594aaf91d6e4ff1f28de58f6077c3359e980a1074fabe195d35d3fb8ee"}
Jul 14 02:02:14.188362 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.188319    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:02:14.189822 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.189801    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f09255-d824-4fed-b5a4-bb594e253ada" containerID="6161a564e7ab8b66840cd5241234da931eda94fce7e7ee1dc193e7cabc882e96" exitCode=0
Jul 14 02:02:14.189969 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.189875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerDied","Data":"6161a564e7ab8b66840cd5241234da931eda94fce7e7ee1dc193e7cabc882e96"}
Jul 14 02:02:14.204772 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.204753    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:02:14.214153 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.214101    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" podStartSLOduration=8.151773029 podStartE2EDuration="28.214088444s" totalImagesPullingTime="20.062315415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:14.21264688 +0000 UTC m=+27.885521690" watchObservedRunningTime="2026-07-14 02:02:14.214088444 +0000 UTC m=+27.886963244"
Jul 14 02:02:14.886590 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.886550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:14.887023 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:14.886595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:14.887023 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:14.886666    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:14.887023 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:14.886774    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:15.192116 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.192058    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 02:02:15.192533 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.192507    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:02:15.207087 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.207056    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:02:15.743640 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.743607    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mfct5"]
Jul 14 02:02:15.743801 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.743738    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:15.743846 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:15.743816    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:15.749662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.749615    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-grzbf"]
Jul 14 02:02:15.749788 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.749717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:15.750215 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:15.749823    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:15.750338 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.750297    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmhpt"]
Jul 14 02:02:15.750400 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:15.750387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:15.750494 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:15.750475    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:16.195640 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:16.195605    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f09255-d824-4fed-b5a4-bb594e253ada" containerID="bdaca5f19eb3235d708baa22fc615ca6350a2ebeb477c63a1efc91b89226a344" exitCode=0
Jul 14 02:02:16.196113 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:16.195687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerDied","Data":"bdaca5f19eb3235d708baa22fc615ca6350a2ebeb477c63a1efc91b89226a344"}
Jul 14 02:02:16.196113 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:16.195790    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 02:02:17.118946 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:17.118911    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:02:17.885969 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:17.885804    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:17.886341 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:17.885819    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:17.886341 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:17.886124    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:17.886341 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:17.886038    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:17.886341 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:17.885819    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:17.886341 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:17.886218    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:18.203963 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:18.203882    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f09255-d824-4fed-b5a4-bb594e253ada" containerID="21136ec8442f65895823282ef76c04d831a6fdc00288e6ee70846ef331d8de51" exitCode=0
Jul 14 02:02:18.203963 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:18.203942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerDied","Data":"21136ec8442f65895823282ef76c04d831a6fdc00288e6ee70846ef331d8de51"}
Jul 14 02:02:18.219749 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:18.219710    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp" podUID="1dc1b087-17d0-42b5-affa-a6eff7db68f4" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 14 02:02:19.558019 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:19.557896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:19.558019 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:19.557946    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:19.558019 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:19.558002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558089    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558106    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558121    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558144    2584 projected.go:196] Error preparing data for projected volume kube-api-access-gwzv6 for pod openshift-network-diagnostics/network-check-target-grzbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558089    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558203    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret podName:adce1978-62d1-4533-a4b9-b8a4703092aa nodeName:}" failed. No retries permitted until 2026-07-14 02:02:51.558180074 +0000 UTC m=+65.231054854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret") pod "global-pull-secret-syncer-mfct5" (UID: "adce1978-62d1-4533-a4b9-b8a4703092aa") : object "kube-system"/"original-pull-secret" not registered
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558253    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs podName:a296f1cc-563f-4feb-96fe-c4c56ef944b9 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:51.558214935 +0000 UTC m=+65.231089716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs") pod "network-metrics-daemon-nmhpt" (UID: "a296f1cc-563f-4feb-96fe-c4c56ef944b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 02:02:19.558679 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.558269    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6 podName:e7231f7f-4e33-4571-b3ce-0d3f15a44a61 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:51.558261428 +0000 UTC m=+65.231136206 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwzv6" (UniqueName: "kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6") pod "network-check-target-grzbf" (UID: "e7231f7f-4e33-4571-b3ce-0d3f15a44a61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 02:02:19.885843 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:19.885814    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:19.885843 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:19.885831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:19.886074 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.885929    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:19.886074 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:19.885967    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:19.886074 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.886055    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:19.886303 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:19.886161    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:21.886062 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:21.886031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:21.886062 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:21.886055    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:21.886759 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:21.886078    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:21.886759 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:21.886186    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmhpt" podUID="a296f1cc-563f-4feb-96fe-c4c56ef944b9"
Jul 14 02:02:21.886759 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:21.886327    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfct5" podUID="adce1978-62d1-4533-a4b9-b8a4703092aa"
Jul 14 02:02:21.886759 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:21.886417    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grzbf" podUID="e7231f7f-4e33-4571-b3ce-0d3f15a44a61"
Jul 14 02:02:22.120228 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.120207    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-94.ec2.internal" event="NodeReady"
Jul 14 02:02:22.120351 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.120333    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 02:02:22.162526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.162502    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bs7sk"]
Jul 14 02:02:22.164707 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.164687    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.165431 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.165404    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zplpn"]
Jul 14 02:02:22.167332 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.167315    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.167809 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.167783    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 02:02:22.167891 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.167843    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qrrwt\""
Jul 14 02:02:22.168170 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.168125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 02:02:22.170703 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.170683    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 02:02:22.171029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.170950    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 02:02:22.171598 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.171576    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jlvpc\""
Jul 14 02:02:22.171598 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.171590    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 02:02:22.180739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.180645    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bs7sk"]
Jul 14 02:02:22.180739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.180707    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zplpn"]
Jul 14 02:02:22.215573 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.215544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerStarted","Data":"d3d937cc1503f8c6c3875e5612e3263b0548647e42c60c34e56eee4ead8a2607"}
Jul 14 02:02:22.277950 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.277928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e3dfd5-7708-4989-9a82-fc5c29dc0346-cert\") pod \"ingress-canary-zplpn\" (UID: \"34e3dfd5-7708-4989-9a82-fc5c29dc0346\") " pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.278039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.277961    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhrb\" (UniqueName: \"kubernetes.io/projected/34e3dfd5-7708-4989-9a82-fc5c29dc0346-kube-api-access-bwhrb\") pod \"ingress-canary-zplpn\" (UID: \"34e3dfd5-7708-4989-9a82-fc5c29dc0346\") " pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.278039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.277979    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3215752b-e593-47c7-931b-1497ac32ec50-tmp-dir\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.278039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.278024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cdvn\" (UniqueName: \"kubernetes.io/projected/3215752b-e593-47c7-931b-1497ac32ec50-kube-api-access-8cdvn\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.278150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.278072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.278150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.278120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3215752b-e593-47c7-931b-1497ac32ec50-config-volume\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.379301 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3215752b-e593-47c7-931b-1497ac32ec50-config-volume\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.379301 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e3dfd5-7708-4989-9a82-fc5c29dc0346-cert\") pod \"ingress-canary-zplpn\" (UID: \"34e3dfd5-7708-4989-9a82-fc5c29dc0346\") " pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.379492 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhrb\" (UniqueName: \"kubernetes.io/projected/34e3dfd5-7708-4989-9a82-fc5c29dc0346-kube-api-access-bwhrb\") pod \"ingress-canary-zplpn\" (UID: \"34e3dfd5-7708-4989-9a82-fc5c29dc0346\") " pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.379492 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3215752b-e593-47c7-931b-1497ac32ec50-tmp-dir\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.379492 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cdvn\" (UniqueName: \"kubernetes.io/projected/3215752b-e593-47c7-931b-1497ac32ec50-kube-api-access-8cdvn\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.379642 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.379693 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:22.379668    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 02:02:22.379742 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:22.379731    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls podName:3215752b-e593-47c7-931b-1497ac32ec50 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:22.879713339 +0000 UTC m=+36.552588117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls") pod "dns-default-bs7sk" (UID: "3215752b-e593-47c7-931b-1497ac32ec50") : secret "dns-default-metrics-tls" not found
Jul 14 02:02:22.379802 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3215752b-e593-47c7-931b-1497ac32ec50-tmp-dir\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.379896 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.379874    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3215752b-e593-47c7-931b-1497ac32ec50-config-volume\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.383438 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.383423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e3dfd5-7708-4989-9a82-fc5c29dc0346-cert\") pod \"ingress-canary-zplpn\" (UID: \"34e3dfd5-7708-4989-9a82-fc5c29dc0346\") " pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.388273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.388251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cdvn\" (UniqueName: \"kubernetes.io/projected/3215752b-e593-47c7-931b-1497ac32ec50-kube-api-access-8cdvn\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.388539 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.388523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhrb\" (UniqueName: \"kubernetes.io/projected/34e3dfd5-7708-4989-9a82-fc5c29dc0346-kube-api-access-bwhrb\") pod \"ingress-canary-zplpn\" (UID: \"34e3dfd5-7708-4989-9a82-fc5c29dc0346\") " pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.483730 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.483702    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zplpn"
Jul 14 02:02:22.630493 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.630252    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zplpn"]
Jul 14 02:02:22.634740 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:22.634716    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e3dfd5_7708_4989_9a82_fc5c29dc0346.slice/crio-3b4337f74a7b6db6b8f1eb97f9b0df914061c009ecbb5a543b057ca93e86e011 WatchSource:0}: Error finding container 3b4337f74a7b6db6b8f1eb97f9b0df914061c009ecbb5a543b057ca93e86e011: Status 404 returned error can't find the container with id 3b4337f74a7b6db6b8f1eb97f9b0df914061c009ecbb5a543b057ca93e86e011
Jul 14 02:02:22.882807 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:22.882776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:22.882971 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:22.882909    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 02:02:22.882971 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:22.882958    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls podName:3215752b-e593-47c7-931b-1497ac32ec50 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:23.88294582 +0000 UTC m=+37.555820601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls") pod "dns-default-bs7sk" (UID: "3215752b-e593-47c7-931b-1497ac32ec50") : secret "dns-default-metrics-tls" not found
Jul 14 02:02:23.219833 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.219757    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zplpn" event={"ID":"34e3dfd5-7708-4989-9a82-fc5c29dc0346","Type":"ContainerStarted","Data":"3b4337f74a7b6db6b8f1eb97f9b0df914061c009ecbb5a543b057ca93e86e011"}
Jul 14 02:02:23.222853 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.222824    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f09255-d824-4fed-b5a4-bb594e253ada" containerID="d3d937cc1503f8c6c3875e5612e3263b0548647e42c60c34e56eee4ead8a2607" exitCode=0
Jul 14 02:02:23.222990 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.222866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerDied","Data":"d3d937cc1503f8c6c3875e5612e3263b0548647e42c60c34e56eee4ead8a2607"}
Jul 14 02:02:23.886275 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.886240    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:23.886450 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.886246    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:23.886450 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.886246    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:23.889279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889233    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 02:02:23.889405 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889286    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 02:02:23.889405 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:23.889405 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889385    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 02:02:23.889557 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889233    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrlfg\""
Jul 14 02:02:23.889557 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889524    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-klxhd\""
Jul 14 02:02:23.889656 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:23.889563    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 02:02:23.889656 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:23.889617    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls podName:3215752b-e593-47c7-931b-1497ac32ec50 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:25.88959995 +0000 UTC m=+39.562474742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls") pod "dns-default-bs7sk" (UID: "3215752b-e593-47c7-931b-1497ac32ec50") : secret "dns-default-metrics-tls" not found
Jul 14 02:02:23.889807 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:23.889787    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 02:02:24.227641 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:24.227567    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f09255-d824-4fed-b5a4-bb594e253ada" containerID="9f3dd2f759736b92b4abc8fe0bb7778935b1ee2125c6711f465ba17a596715a1" exitCode=0
Jul 14 02:02:24.227641 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:24.227602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerDied","Data":"9f3dd2f759736b92b4abc8fe0bb7778935b1ee2125c6711f465ba17a596715a1"}
Jul 14 02:02:24.677784 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:24.677757    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lddfv_81f850e6-9a94-4e26-9880-39802130d134/dns-node-resolver/0.log"
Jul 14 02:02:25.232285 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:25.232251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" event={"ID":"53f09255-d824-4fed-b5a4-bb594e253ada","Type":"ContainerStarted","Data":"6bfbea00dd95426788bad5bd71e4000fe6726d518c84a7b1d855aae1c46aabff"}
Jul 14 02:02:25.264385 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:25.264116    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rzbxv" podStartSLOduration=9.281224775 podStartE2EDuration="39.264096653s" totalImagesPullingTime="29.982871878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:25.261895994 +0000 UTC m=+38.934770794" watchObservedRunningTime="2026-07-14 02:02:25.264096653 +0000 UTC m=+38.936971454"
Jul 14 02:02:25.680382 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:25.680354    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkztz_a70b3316-2d49-4928-9428-88d957426bf7/node-ca/0.log"
Jul 14 02:02:25.903725 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:25.903696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:25.903914 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:25.903848    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 02:02:25.903914 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:02:25.903904    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls podName:3215752b-e593-47c7-931b-1497ac32ec50 nodeName:}" failed. No retries permitted until 2026-07-14 02:02:29.903891587 +0000 UTC m=+43.576766366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls") pod "dns-default-bs7sk" (UID: "3215752b-e593-47c7-931b-1497ac32ec50") : secret "dns-default-metrics-tls" not found
Jul 14 02:02:26.236070 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:26.235983    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zplpn" event={"ID":"34e3dfd5-7708-4989-9a82-fc5c29dc0346","Type":"ContainerStarted","Data":"264bcf93a1f3424ef1b5e34038a863c9e3a15215fce14103612fab5a57eb0d2c"}
Jul 14 02:02:26.253279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:26.253238    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zplpn" podStartSLOduration=1.429464734 podStartE2EDuration="4.253221954s" totalImagesPullingTime="2.82375722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:26.252428329 +0000 UTC m=+39.925303141" watchObservedRunningTime="2026-07-14 02:02:26.253221954 +0000 UTC m=+39.926096754"
Jul 14 02:02:26.677224 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:26.677194    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zplpn_34e3dfd5-7708-4989-9a82-fc5c29dc0346/serve-healthcheck-canary/0.log"
Jul 14 02:02:29.933193 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:29.933162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:29.937077 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:29.937058    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3215752b-e593-47c7-931b-1497ac32ec50-metrics-tls\") pod \"dns-default-bs7sk\" (UID: \"3215752b-e593-47c7-931b-1497ac32ec50\") " pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:29.977111 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:29.977083    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:30.118604 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:30.118577    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bs7sk"]
Jul 14 02:02:30.121717 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:30.121690    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3215752b_e593_47c7_931b_1497ac32ec50.slice/crio-e6ed8f67ecfee48e1eeb0fd9276d2f7b3f2686bacb1d4b3ad1e0a61402c99989 WatchSource:0}: Error finding container e6ed8f67ecfee48e1eeb0fd9276d2f7b3f2686bacb1d4b3ad1e0a61402c99989: Status 404 returned error can't find the container with id e6ed8f67ecfee48e1eeb0fd9276d2f7b3f2686bacb1d4b3ad1e0a61402c99989
Jul 14 02:02:30.246636 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:30.246565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bs7sk" event={"ID":"3215752b-e593-47c7-931b-1497ac32ec50","Type":"ContainerStarted","Data":"e6ed8f67ecfee48e1eeb0fd9276d2f7b3f2686bacb1d4b3ad1e0a61402c99989"}
Jul 14 02:02:33.254961 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:33.254928    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bs7sk" event={"ID":"3215752b-e593-47c7-931b-1497ac32ec50","Type":"ContainerStarted","Data":"6012aa2d7937cbaeda32529e815bf3f7f7fc527b9a48223b4c8a0838b807c605"}
Jul 14 02:02:33.255467 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:33.254978    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bs7sk" event={"ID":"3215752b-e593-47c7-931b-1497ac32ec50","Type":"ContainerStarted","Data":"51b9ea14d088ee1138f960250f3f5f8abfd436c01e074044b285c50bdddc964c"}
Jul 14 02:02:33.255467 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:33.255064    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:33.275000 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:33.274961    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bs7sk" podStartSLOduration=8.895489897000001 podStartE2EDuration="11.27494658s" totalImagesPullingTime="2.379456683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:33.273282136 +0000 UTC m=+46.946156939" watchObservedRunningTime="2026-07-14 02:02:33.27494658 +0000 UTC m=+46.947821379"
Jul 14 02:02:40.902252 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.902213    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"]
Jul 14 02:02:40.916721 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.916696    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"]
Jul 14 02:02:40.916871 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.916790    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:40.919611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.919587    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 02:02:40.920675 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.920653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2x8vh\""
Jul 14 02:02:40.920857 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.920721    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 02:02:40.920857 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.920723    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 02:02:40.920857 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.920668    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 14 02:02:40.934869 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.934850    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"]
Jul 14 02:02:40.934979 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.934865    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:40.937514 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.937494    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 14 02:02:40.937639 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.937537    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 14 02:02:40.937639 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.937549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 14 02:02:40.937791 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.937775    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 14 02:02:40.951325 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.951308    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"]
Jul 14 02:02:40.951325 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.951320    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:40.951481 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.951342    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"]
Jul 14 02:02:40.951481 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.951361    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"]
Jul 14 02:02:40.955687 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:40.955673    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 14 02:02:41.006049 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.006026    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6cd6b"]
Jul 14 02:02:41.007893 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.007869    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.011378 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.011357    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe253756-b9f1-47ac-a720-6c65f73bbe2e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz\" (UID: \"fe253756-b9f1-47ac-a720-6c65f73bbe2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.011473 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.011388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bnw\" (UniqueName: \"kubernetes.io/projected/fe253756-b9f1-47ac-a720-6c65f73bbe2e-kube-api-access-s9bnw\") pod \"managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz\" (UID: \"fe253756-b9f1-47ac-a720-6c65f73bbe2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.011821 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.011798    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 02:02:41.011882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.011820    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 02:02:41.011882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.011829    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 02:02:41.011882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.011858    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 02:02:41.012085 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.012063    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xfgbl\""
Jul 14 02:02:41.023838 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.023809    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6cd6b"]
Jul 14 02:02:41.112719 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe253756-b9f1-47ac-a720-6c65f73bbe2e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz\" (UID: \"fe253756-b9f1-47ac-a720-6c65f73bbe2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.112893 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112731    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prlnl\" (UniqueName: \"kubernetes.io/projected/53b2d150-03b0-425d-9f08-ab01b238d425-kube-api-access-prlnl\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.112893 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe2c527-2272-42c3-b65d-c4d4af45c210-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.112893 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm255\" (UniqueName: \"kubernetes.io/projected/1fe2c527-2272-42c3-b65d-c4d4af45c210-kube-api-access-jm255\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.112893 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.112893 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bnw\" (UniqueName: \"kubernetes.io/projected/fe253756-b9f1-47ac-a720-6c65f73bbe2e-kube-api-access-s9bnw\") pod \"managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz\" (UID: \"fe253756-b9f1-47ac-a720-6c65f73bbe2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.113079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53b2d150-03b0-425d-9f08-ab01b238d425-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.113079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45126911-b0a5-442a-a1f0-b06b46c27730-tmp\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.113079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.112967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53b2d150-03b0-425d-9f08-ab01b238d425-crio-socket\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.113079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113001    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.113079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b2d150-03b0-425d-9f08-ab01b238d425-data-volume\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.113296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-ca\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.113296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/45126911-b0a5-442a-a1f0-b06b46c27730-klusterlet-config\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.113296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-hub\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.113296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q477\" (UniqueName: \"kubernetes.io/projected/45126911-b0a5-442a-a1f0-b06b46c27730-kube-api-access-9q477\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.113296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.113204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53b2d150-03b0-425d-9f08-ab01b238d425-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.115207 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.115192    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe253756-b9f1-47ac-a720-6c65f73bbe2e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz\" (UID: \"fe253756-b9f1-47ac-a720-6c65f73bbe2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.122551 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.122532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bnw\" (UniqueName: \"kubernetes.io/projected/fe253756-b9f1-47ac-a720-6c65f73bbe2e-kube-api-access-s9bnw\") pod \"managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz\" (UID: \"fe253756-b9f1-47ac-a720-6c65f73bbe2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.214213 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53b2d150-03b0-425d-9f08-ab01b238d425-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.214213 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45126911-b0a5-442a-a1f0-b06b46c27730-tmp\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.214213 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53b2d150-03b0-425d-9f08-ab01b238d425-crio-socket\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.214213 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b2d150-03b0-425d-9f08-ab01b238d425-data-volume\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-ca\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/45126911-b0a5-442a-a1f0-b06b46c27730-klusterlet-config\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-hub\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q477\" (UniqueName: \"kubernetes.io/projected/45126911-b0a5-442a-a1f0-b06b46c27730-kube-api-access-9q477\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53b2d150-03b0-425d-9f08-ab01b238d425-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214387    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53b2d150-03b0-425d-9f08-ab01b238d425-crio-socket\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prlnl\" (UniqueName: \"kubernetes.io/projected/53b2d150-03b0-425d-9f08-ab01b238d425-kube-api-access-prlnl\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe2c527-2272-42c3-b65d-c4d4af45c210-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214464    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm255\" (UniqueName: \"kubernetes.io/projected/1fe2c527-2272-42c3-b65d-c4d4af45c210-kube-api-access-jm255\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.214506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.215026 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b2d150-03b0-425d-9f08-ab01b238d425-data-volume\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.215026 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.214572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45126911-b0a5-442a-a1f0-b06b46c27730-tmp\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.215408 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.215187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53b2d150-03b0-425d-9f08-ab01b238d425-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.215510 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.215450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe2c527-2272-42c3-b65d-c4d4af45c210-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.216888 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.216863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53b2d150-03b0-425d-9f08-ab01b238d425-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.217219 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.217199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-ca\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.217314 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.217299    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.217549 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.217521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/45126911-b0a5-442a-a1f0-b06b46c27730-klusterlet-config\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.217894 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.217875    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.217938 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.217894    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1fe2c527-2272-42c3-b65d-c4d4af45c210-hub\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.224337 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.224315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm255\" (UniqueName: \"kubernetes.io/projected/1fe2c527-2272-42c3-b65d-c4d4af45c210-kube-api-access-jm255\") pod \"cluster-proxy-proxy-agent-7f49b89c97-szr9x\" (UID: \"1fe2c527-2272-42c3-b65d-c4d4af45c210\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.226202 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.226184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q477\" (UniqueName: \"kubernetes.io/projected/45126911-b0a5-442a-a1f0-b06b46c27730-kube-api-access-9q477\") pod \"klusterlet-addon-workmgr-97cd46854-4wpzx\" (UID: \"45126911-b0a5-442a-a1f0-b06b46c27730\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.229218 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.229198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prlnl\" (UniqueName: \"kubernetes.io/projected/53b2d150-03b0-425d-9f08-ab01b238d425-kube-api-access-prlnl\") pod \"insights-runtime-extractor-6cd6b\" (UID: \"53b2d150-03b0-425d-9f08-ab01b238d425\") " pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.233001 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.232982    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"
Jul 14 02:02:41.242698 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.242673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"
Jul 14 02:02:41.271644 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.271614    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:41.316675 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.316642    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6cd6b"
Jul 14 02:02:41.372908 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.372852    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz"]
Jul 14 02:02:41.403179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.403115    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x"]
Jul 14 02:02:41.405929 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:41.405899    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe2c527_2272_42c3_b65d_c4d4af45c210.slice/crio-dccce16fb2a124ca829857ee06850f87d37f35130575b30327b460fce0fed931 WatchSource:0}: Error finding container dccce16fb2a124ca829857ee06850f87d37f35130575b30327b460fce0fed931: Status 404 returned error can't find the container with id dccce16fb2a124ca829857ee06850f87d37f35130575b30327b460fce0fed931
Jul 14 02:02:41.418809 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.418782    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"]
Jul 14 02:02:41.422002 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:41.421973    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45126911_b0a5_442a_a1f0_b06b46c27730.slice/crio-3054fb56010086eb5e9d3bb7ad4461833e84b65b8fbd6feebfac353d73f8404b WatchSource:0}: Error finding container 3054fb56010086eb5e9d3bb7ad4461833e84b65b8fbd6feebfac353d73f8404b: Status 404 returned error can't find the container with id 3054fb56010086eb5e9d3bb7ad4461833e84b65b8fbd6feebfac353d73f8404b
Jul 14 02:02:41.456668 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:41.456636    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6cd6b"]
Jul 14 02:02:41.459172 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:41.459122    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b2d150_03b0_425d_9f08_ab01b238d425.slice/crio-d0bfc868438d2cf1a8157f46b86c75b7186a7ba3f6a6dc589fac51f3840bb45d WatchSource:0}: Error finding container d0bfc868438d2cf1a8157f46b86c75b7186a7ba3f6a6dc589fac51f3840bb45d: Status 404 returned error can't find the container with id d0bfc868438d2cf1a8157f46b86c75b7186a7ba3f6a6dc589fac51f3840bb45d
Jul 14 02:02:42.281927 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:42.281844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz" event={"ID":"fe253756-b9f1-47ac-a720-6c65f73bbe2e","Type":"ContainerStarted","Data":"d257cdc2c840793ee67ebe9db78f5e3693f0f7d20ff9a2b8f5ea9e042c8512e9"}
Jul 14 02:02:42.289971 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:42.289871    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cd6b" event={"ID":"53b2d150-03b0-425d-9f08-ab01b238d425","Type":"ContainerStarted","Data":"d0bfc868438d2cf1a8157f46b86c75b7186a7ba3f6a6dc589fac51f3840bb45d"}
Jul 14 02:02:42.308535 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:42.308497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx" event={"ID":"45126911-b0a5-442a-a1f0-b06b46c27730","Type":"ContainerStarted","Data":"3054fb56010086eb5e9d3bb7ad4461833e84b65b8fbd6feebfac353d73f8404b"}
Jul 14 02:02:42.313737 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:42.313696    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x" event={"ID":"1fe2c527-2272-42c3-b65d-c4d4af45c210","Type":"ContainerStarted","Data":"dccce16fb2a124ca829857ee06850f87d37f35130575b30327b460fce0fed931"}
Jul 14 02:02:43.260866 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:43.260542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bs7sk"
Jul 14 02:02:48.213658 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:48.213629    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrcp"
Jul 14 02:02:51.350390 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.350336    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz" event={"ID":"fe253756-b9f1-47ac-a720-6c65f73bbe2e","Type":"ContainerStarted","Data":"79a84f18a25e486df8398b5875152c45c4aa6111844d75d83b092ab2690d5cad"}
Jul 14 02:02:51.352228 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.352200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cd6b" event={"ID":"53b2d150-03b0-425d-9f08-ab01b238d425","Type":"ContainerStarted","Data":"9ca724c036e09b1fe89348c7aba80f6604db54cee5d4f018c3e5ccc359d4df95"}
Jul 14 02:02:51.352371 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.352260    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cd6b" event={"ID":"53b2d150-03b0-425d-9f08-ab01b238d425","Type":"ContainerStarted","Data":"2cf15289fa1e670b5f258903a5765937d5134709b8099ccde2b998b0859747fa"}
Jul 14 02:02:51.353806 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.353787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx" event={"ID":"45126911-b0a5-442a-a1f0-b06b46c27730","Type":"ContainerStarted","Data":"355b7d6b07efde5fff602e9fcd0dd61ed7b8f82b1c0face0edfcc3c20d351bd9"}
Jul 14 02:02:51.354004 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.353986    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:51.356229 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.356210    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx"
Jul 14 02:02:51.356350 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.356287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x" event={"ID":"1fe2c527-2272-42c3-b65d-c4d4af45c210","Type":"ContainerStarted","Data":"5ff39e1bd725aba72a4aef35ef6df322f5439d87c20ffb2bbd88f99d376d4e2d"}
Jul 14 02:02:51.356350 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.356323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x" event={"ID":"1fe2c527-2272-42c3-b65d-c4d4af45c210","Type":"ContainerStarted","Data":"3ca872c8e6bbd21e5fc625d535bc8c335c299a33ca4160db508b21ffaacf3079"}
Jul 14 02:02:51.356350 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.356342    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x" event={"ID":"1fe2c527-2272-42c3-b65d-c4d4af45c210","Type":"ContainerStarted","Data":"e0de343340f6eb5f512c9d2087f389fa8cd76c96f3c93b3dfe680215110d3161"}
Jul 14 02:02:51.367330 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.367281    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd4cbbdcd-cj9lz" podStartSLOduration=2.489241906 podStartE2EDuration="11.367262795s" totalImagesPullingTime="8.878020889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:51.365765624 +0000 UTC m=+65.038640425" watchObservedRunningTime="2026-07-14 02:02:51.367262795 +0000 UTC m=+65.040137596"
Jul 14 02:02:51.384882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.384722    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f49b89c97-szr9x" podStartSLOduration=2.5280463859999998 podStartE2EDuration="11.384705291s" totalImagesPullingTime="8.856658905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:51.384326492 +0000 UTC m=+65.057201299" watchObservedRunningTime="2026-07-14 02:02:51.384705291 +0000 UTC m=+65.057580092"
Jul 14 02:02:51.399795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.399639    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97cd46854-4wpzx" podStartSLOduration=2.558335526 podStartE2EDuration="11.399623252s" totalImagesPullingTime="8.841287726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:51.399121175 +0000 UTC m=+65.071995978" watchObservedRunningTime="2026-07-14 02:02:51.399623252 +0000 UTC m=+65.072498052"
Jul 14 02:02:51.600919 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.600840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:51.601092 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.600920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:51.601092 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.600970    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:51.604009 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.603985    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 02:02:51.604009 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.604002    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 02:02:51.604213 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.604038    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 02:02:51.613767 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.613744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adce1978-62d1-4533-a4b9-b8a4703092aa-original-pull-secret\") pod \"global-pull-secret-syncer-mfct5\" (UID: \"adce1978-62d1-4533-a4b9-b8a4703092aa\") " pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:51.613902 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.613803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a296f1cc-563f-4feb-96fe-c4c56ef944b9-metrics-certs\") pod \"network-metrics-daemon-nmhpt\" (UID: \"a296f1cc-563f-4feb-96fe-c4c56ef944b9\") " pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:51.614200 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.614184    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 02:02:51.624671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.624650    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzv6\" (UniqueName: \"kubernetes.io/projected/e7231f7f-4e33-4571-b3ce-0d3f15a44a61-kube-api-access-gwzv6\") pod \"network-check-target-grzbf\" (UID: \"e7231f7f-4e33-4571-b3ce-0d3f15a44a61\") " pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:51.798688 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.798656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfct5"
Jul 14 02:02:51.807743 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.807720    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrlfg\""
Jul 14 02:02:51.811029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.811000    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-klxhd\""
Jul 14 02:02:51.816083 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.816036    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:51.818974 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.818953    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmhpt"
Jul 14 02:02:51.964909 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:51.963830    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mfct5"]
Jul 14 02:02:51.968702 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:51.968625    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadce1978_62d1_4533_a4b9_b8a4703092aa.slice/crio-dfbf2665a95e60c94b0bf9807124539a3641127dbc55d4a391cf7df1b45df365 WatchSource:0}: Error finding container dfbf2665a95e60c94b0bf9807124539a3641127dbc55d4a391cf7df1b45df365: Status 404 returned error can't find the container with id dfbf2665a95e60c94b0bf9807124539a3641127dbc55d4a391cf7df1b45df365
Jul 14 02:02:52.193850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:52.193818    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-grzbf"]
Jul 14 02:02:52.196709 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:52.196684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmhpt"]
Jul 14 02:02:52.197121 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:52.197072    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7231f7f_4e33_4571_b3ce_0d3f15a44a61.slice/crio-0144871cc379b677529492154cb0d246c1d4d78d81fdfb3632728d65ee1f4c25 WatchSource:0}: Error finding container 0144871cc379b677529492154cb0d246c1d4d78d81fdfb3632728d65ee1f4c25: Status 404 returned error can't find the container with id 0144871cc379b677529492154cb0d246c1d4d78d81fdfb3632728d65ee1f4c25
Jul 14 02:02:52.199404 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:52.199379    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda296f1cc_563f_4feb_96fe_c4c56ef944b9.slice/crio-72530cc41f372202fe2e5b3b6c32d3a9e9b8753442eb53158f8d62f42cb18c9a WatchSource:0}: Error finding container 72530cc41f372202fe2e5b3b6c32d3a9e9b8753442eb53158f8d62f42cb18c9a: Status 404 returned error can't find the container with id 72530cc41f372202fe2e5b3b6c32d3a9e9b8753442eb53158f8d62f42cb18c9a
Jul 14 02:02:52.365218 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:52.365175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-grzbf" event={"ID":"e7231f7f-4e33-4571-b3ce-0d3f15a44a61","Type":"ContainerStarted","Data":"0144871cc379b677529492154cb0d246c1d4d78d81fdfb3632728d65ee1f4c25"}
Jul 14 02:02:52.366881 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:52.366853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mfct5" event={"ID":"adce1978-62d1-4533-a4b9-b8a4703092aa","Type":"ContainerStarted","Data":"dfbf2665a95e60c94b0bf9807124539a3641127dbc55d4a391cf7df1b45df365"}
Jul 14 02:02:52.368651 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:52.368618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmhpt" event={"ID":"a296f1cc-563f-4feb-96fe-c4c56ef944b9","Type":"ContainerStarted","Data":"72530cc41f372202fe2e5b3b6c32d3a9e9b8753442eb53158f8d62f42cb18c9a"}
Jul 14 02:02:54.378993 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:54.378951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cd6b" event={"ID":"53b2d150-03b0-425d-9f08-ab01b238d425","Type":"ContainerStarted","Data":"378581fae92780b5aa4ed300febbc48168dec8871e066660297ffb9ee6f4e5a4"}
Jul 14 02:02:54.401420 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:54.401364    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6cd6b" podStartSLOduration=2.413486106 podStartE2EDuration="14.401340923s" totalImagesPullingTime="11.987854817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:02:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:54.400270168 +0000 UTC m=+68.073144969" watchObservedRunningTime="2026-07-14 02:02:54.401340923 +0000 UTC m=+68.074215725"
Jul 14 02:02:55.386289 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:55.386257    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmhpt" event={"ID":"a296f1cc-563f-4feb-96fe-c4c56ef944b9","Type":"ContainerStarted","Data":"591dc6d82f6acea3bda67b6b83a1ef199379af79133e7ef8a9157201023e9dcb"}
Jul 14 02:02:55.386710 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:55.386315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmhpt" event={"ID":"a296f1cc-563f-4feb-96fe-c4c56ef944b9","Type":"ContainerStarted","Data":"3f3045a3e382e4b8486cd5a8455c01e5a37566570b0dc5c146107fd1f79a73c4"}
Jul 14 02:02:55.404281 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:55.404232    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nmhpt" podStartSLOduration=67.150104494 podStartE2EDuration="1m9.404210316s" totalImagesPullingTime="2.254105822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:55.40274034 +0000 UTC m=+69.075615144" watchObservedRunningTime="2026-07-14 02:02:55.404210316 +0000 UTC m=+69.077085117"
Jul 14 02:02:57.394065 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:57.393995    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mfct5" event={"ID":"adce1978-62d1-4533-a4b9-b8a4703092aa","Type":"ContainerStarted","Data":"01fbc41ea7f13e2629989be233b35c76b4ecd48461ba67721848854c456a32c7"}
Jul 14 02:02:57.395464 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:57.395434    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-grzbf" event={"ID":"e7231f7f-4e33-4571-b3ce-0d3f15a44a61","Type":"ContainerStarted","Data":"473b23b9bfedf64eb0421d6933ca423549a63b667d025ccf33bbe72948489d79"}
Jul 14 02:02:57.395592 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:57.395554    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:02:57.408571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:57.408534    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mfct5" podStartSLOduration=65.127002223 podStartE2EDuration="1m10.408520608s" totalImagesPullingTime="5.281518385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:57.40703566 +0000 UTC m=+71.079910461" watchObservedRunningTime="2026-07-14 02:02:57.408520608 +0000 UTC m=+71.081395407"
Jul 14 02:02:57.420795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:57.420758    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-grzbf" podStartSLOduration=65.368029628 podStartE2EDuration="1m10.420745768s" totalImagesPullingTime="5.05271614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:02:57.420047647 +0000 UTC m=+71.092922449" watchObservedRunningTime="2026-07-14 02:02:57.420745768 +0000 UTC m=+71.093620567"
Jul 14 02:02:58.636079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.636046    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9gc2g"]
Jul 14 02:02:58.638227 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.638212    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.641040 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.641013    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 02:02:58.641040 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.641023    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 02:02:58.641040 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.641017    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pgskw\""
Jul 14 02:02:58.641322 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.641023    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 02:02:58.642215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.642194    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 02:02:58.642327 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.642196    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 02:02:58.642327 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.642237    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 02:02:58.756601 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-textfile\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756639    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-tls\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756661    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756681    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14de5b20-d544-4d51-8698-5cb1fd56b692-metrics-client-ca\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bhd\" (UniqueName: \"kubernetes.io/projected/14de5b20-d544-4d51-8698-5cb1fd56b692-kube-api-access-n2bhd\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-wtmp\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756976 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756801    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-root\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.756976 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.756820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-sys\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858099 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-root\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858103    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-sys\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-textfile\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858207    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-root\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-tls\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858207    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-sys\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14de5b20-d544-4d51-8698-5cb1fd56b692-metrics-client-ca\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bhd\" (UniqueName: \"kubernetes.io/projected/14de5b20-d544-4d51-8698-5cb1fd56b692-kube-api-access-n2bhd\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858690 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858332    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-wtmp\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858690 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-wtmp\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858690 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858575    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-textfile\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.858934 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.859008 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.858915    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14de5b20-d544-4d51-8698-5cb1fd56b692-metrics-client-ca\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.860616 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.860591    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.860721 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.860687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14de5b20-d544-4d51-8698-5cb1fd56b692-node-exporter-tls\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.867192 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.867168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bhd\" (UniqueName: \"kubernetes.io/projected/14de5b20-d544-4d51-8698-5cb1fd56b692-kube-api-access-n2bhd\") pod \"node-exporter-9gc2g\" (UID: \"14de5b20-d544-4d51-8698-5cb1fd56b692\") " pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.947183 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:58.947096    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gc2g"
Jul 14 02:02:58.957762 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:02:58.957733    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14de5b20_d544_4d51_8698_5cb1fd56b692.slice/crio-4029c8b8f2f4bee350bb864b18a4a2ca3d5f83cfddf93a3b8661a9360416e258 WatchSource:0}: Error finding container 4029c8b8f2f4bee350bb864b18a4a2ca3d5f83cfddf93a3b8661a9360416e258: Status 404 returned error can't find the container with id 4029c8b8f2f4bee350bb864b18a4a2ca3d5f83cfddf93a3b8661a9360416e258
Jul 14 02:02:59.404512 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:02:59.404477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gc2g" event={"ID":"14de5b20-d544-4d51-8698-5cb1fd56b692","Type":"ContainerStarted","Data":"4029c8b8f2f4bee350bb864b18a4a2ca3d5f83cfddf93a3b8661a9360416e258"}
Jul 14 02:03:00.408684 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:00.408652    2584 generic.go:356] "Generic (PLEG): container finished" podID="14de5b20-d544-4d51-8698-5cb1fd56b692" containerID="ef8fff7a63247243a2bf9eadfb31e09bf5b4f2e17cb08aa5d7e4454165104888" exitCode=0
Jul 14 02:03:00.409070 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:00.408699    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gc2g" event={"ID":"14de5b20-d544-4d51-8698-5cb1fd56b692","Type":"ContainerDied","Data":"ef8fff7a63247243a2bf9eadfb31e09bf5b4f2e17cb08aa5d7e4454165104888"}
Jul 14 02:03:01.414044 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:01.414009    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gc2g" event={"ID":"14de5b20-d544-4d51-8698-5cb1fd56b692","Type":"ContainerStarted","Data":"66f4bff73e5d2f4af179f62e74261bb14323221728383e93401ad68bd6d0bb3b"}
Jul 14 02:03:01.414418 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:01.414058    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gc2g" event={"ID":"14de5b20-d544-4d51-8698-5cb1fd56b692","Type":"ContainerStarted","Data":"fc128ddd8daa344df75e29458ee3cd23b86fa79f2f727df0120499dfd779dee6"}
Jul 14 02:03:08.619930 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.619799    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9gc2g" podStartSLOduration=9.677498284 podStartE2EDuration="10.619782279s" totalImagesPullingTime="877.965682ms" totalInitContainerRuntime="64.318313ms" isStatefulPod=true podCreationTimestamp="2026-07-14 02:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:03:01.450639029 +0000 UTC m=+75.123513831" watchObservedRunningTime="2026-07-14 02:03:08.619782279 +0000 UTC m=+82.292657098"
Jul 14 02:03:08.621770 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.621727    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-n2jm2"]
Jul 14 02:03:08.624940 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.624919    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:03:08.627797 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.627776    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 02:03:08.627797 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.627788    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bbndh\""
Jul 14 02:03:08.627949 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.627821    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 02:03:08.635110 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.635079    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-n2jm2"]
Jul 14 02:03:08.726273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.726245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshbw\" (UniqueName: \"kubernetes.io/projected/e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e-kube-api-access-vshbw\") pod \"downloads-68c9cbb497-n2jm2\" (UID: \"e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e\") " pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:03:08.826896 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.826863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshbw\" (UniqueName: \"kubernetes.io/projected/e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e-kube-api-access-vshbw\") pod \"downloads-68c9cbb497-n2jm2\" (UID: \"e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e\") " pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:03:08.835304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.835281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshbw\" (UniqueName: \"kubernetes.io/projected/e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e-kube-api-access-vshbw\") pod \"downloads-68c9cbb497-n2jm2\" (UID: \"e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e\") " pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:03:08.934032 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:08.934003    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:03:09.068288 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:09.068264    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-n2jm2"]
Jul 14 02:03:09.070506 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:03:09.070470    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b4d3f1_1e96_48f1_935f_b2b9a7f5953e.slice/crio-867e6f89333ec872ca99db5f7cea812f553ceb6a70f5aeb1dd044fa9f54c2459 WatchSource:0}: Error finding container 867e6f89333ec872ca99db5f7cea812f553ceb6a70f5aeb1dd044fa9f54c2459: Status 404 returned error can't find the container with id 867e6f89333ec872ca99db5f7cea812f553ceb6a70f5aeb1dd044fa9f54c2459
Jul 14 02:03:09.443758 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:09.443721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-n2jm2" event={"ID":"e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e","Type":"ContainerStarted","Data":"867e6f89333ec872ca99db5f7cea812f553ceb6a70f5aeb1dd044fa9f54c2459"}
Jul 14 02:03:28.402826 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:28.402790    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-grzbf"
Jul 14 02:03:29.516550 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:29.516437    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-n2jm2" event={"ID":"e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e","Type":"ContainerStarted","Data":"c2c1a145c2d841d2616a05c4fb5eebab27e8316bb2d1a7376effe9adc9136c93"}
Jul 14 02:03:29.516971 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:29.516675    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:03:29.518353 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:29.518325    2584 patch_prober.go:28] interesting pod/downloads-68c9cbb497-n2jm2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body=
Jul 14 02:03:29.518470 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:29.518373    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-n2jm2" podUID="e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused"
Jul 14 02:03:29.532493 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:29.532447    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-n2jm2" podStartSLOduration=1.448268718 podStartE2EDuration="21.532428412s" totalImagesPullingTime="20.084159694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:03:29.531731934 +0000 UTC m=+103.204606737" watchObservedRunningTime="2026-07-14 02:03:29.532428412 +0000 UTC m=+103.205303213"
Jul 14 02:03:30.532123 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:03:30.532083    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-n2jm2"
Jul 14 02:06:46.713403 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:06:46.713376    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 02:07:03.108098 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.108063    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"]
Jul 14 02:07:03.109861 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.109847    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.112556 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.112530    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 14 02:07:03.112556 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.112553    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tprkx\""
Jul 14 02:07:03.112736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.112572    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 14 02:07:03.125748 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.125719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"]
Jul 14 02:07:03.224940 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.224911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtrd\" (UniqueName: \"kubernetes.io/projected/2ca995f1-6173-42bb-bb83-a81fdc421633-kube-api-access-hwtrd\") pod \"cert-manager-operator-controller-manager-55c444bf86-chxz7\" (UID: \"2ca995f1-6173-42bb-bb83-a81fdc421633\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.225102 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.224949    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ca995f1-6173-42bb-bb83-a81fdc421633-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-chxz7\" (UID: \"2ca995f1-6173-42bb-bb83-a81fdc421633\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.326183 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.326151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ca995f1-6173-42bb-bb83-a81fdc421633-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-chxz7\" (UID: \"2ca995f1-6173-42bb-bb83-a81fdc421633\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.326320 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.326235    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtrd\" (UniqueName: \"kubernetes.io/projected/2ca995f1-6173-42bb-bb83-a81fdc421633-kube-api-access-hwtrd\") pod \"cert-manager-operator-controller-manager-55c444bf86-chxz7\" (UID: \"2ca995f1-6173-42bb-bb83-a81fdc421633\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.326548 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.326527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ca995f1-6173-42bb-bb83-a81fdc421633-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-chxz7\" (UID: \"2ca995f1-6173-42bb-bb83-a81fdc421633\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.338267 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.338237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtrd\" (UniqueName: \"kubernetes.io/projected/2ca995f1-6173-42bb-bb83-a81fdc421633-kube-api-access-hwtrd\") pod \"cert-manager-operator-controller-manager-55c444bf86-chxz7\" (UID: \"2ca995f1-6173-42bb-bb83-a81fdc421633\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.418875 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.418848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"
Jul 14 02:07:03.546822 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.546779    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7"]
Jul 14 02:07:03.550395 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:07:03.550368    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca995f1_6173_42bb_bb83_a81fdc421633.slice/crio-049f89e309d70401fe2924192cd8f439ba3b42d1dc77acdf6042a3d7651f0bfb WatchSource:0}: Error finding container 049f89e309d70401fe2924192cd8f439ba3b42d1dc77acdf6042a3d7651f0bfb: Status 404 returned error can't find the container with id 049f89e309d70401fe2924192cd8f439ba3b42d1dc77acdf6042a3d7651f0bfb
Jul 14 02:07:03.553054 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:03.553037    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 02:07:04.183927 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:04.183892    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7" event={"ID":"2ca995f1-6173-42bb-bb83-a81fdc421633","Type":"ContainerStarted","Data":"049f89e309d70401fe2924192cd8f439ba3b42d1dc77acdf6042a3d7651f0bfb"}
Jul 14 02:07:06.193045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:06.193011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7" event={"ID":"2ca995f1-6173-42bb-bb83-a81fdc421633","Type":"ContainerStarted","Data":"33d6947d981980a5f63db666381b8de874b54875ac03b73a0125e0101b813310"}
Jul 14 02:07:06.214680 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:06.214627    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-chxz7" podStartSLOduration=0.773839701 podStartE2EDuration="3.214611227s" totalImagesPullingTime="2.440771526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:07:06.211650962 +0000 UTC m=+319.884525762" watchObservedRunningTime="2026-07-14 02:07:06.214611227 +0000 UTC m=+319.887486026"
Jul 14 02:07:45.342194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.342105    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"]
Jul 14 02:07:45.346211 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.346193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.349564 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.349538    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 02:07:45.350178 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.350161    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 02:07:45.350603 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.350587    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wpb5w\""
Jul 14 02:07:45.356402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.356381    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 02:07:45.378263 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.378241    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 02:07:45.380983 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.380964    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"]
Jul 14 02:07:45.519387 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.519356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44dd7d23-a748-44a0-8cbb-15a010bef950-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.519387 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.519389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchwl\" (UniqueName: \"kubernetes.io/projected/44dd7d23-a748-44a0-8cbb-15a010bef950-kube-api-access-jchwl\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.519618 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.519415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44dd7d23-a748-44a0-8cbb-15a010bef950-webhook-cert\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.519618 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.519516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44dd7d23-a748-44a0-8cbb-15a010bef950-manifests\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.519618 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.519551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44dd7d23-a748-44a0-8cbb-15a010bef950-tmp\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.620694 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.620602    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchwl\" (UniqueName: \"kubernetes.io/projected/44dd7d23-a748-44a0-8cbb-15a010bef950-kube-api-access-jchwl\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.620694 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.620655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44dd7d23-a748-44a0-8cbb-15a010bef950-webhook-cert\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.620694 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.620699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44dd7d23-a748-44a0-8cbb-15a010bef950-manifests\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.620941 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.620717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44dd7d23-a748-44a0-8cbb-15a010bef950-tmp\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.620941 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.620755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44dd7d23-a748-44a0-8cbb-15a010bef950-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.621191 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.621168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44dd7d23-a748-44a0-8cbb-15a010bef950-manifests\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.622984 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.622954    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44dd7d23-a748-44a0-8cbb-15a010bef950-tmp\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.623245 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.623226    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44dd7d23-a748-44a0-8cbb-15a010bef950-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.623317 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.623258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44dd7d23-a748-44a0-8cbb-15a010bef950-webhook-cert\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.629417 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.629395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchwl\" (UniqueName: \"kubernetes.io/projected/44dd7d23-a748-44a0-8cbb-15a010bef950-kube-api-access-jchwl\") pod \"opendatahub-operator-controller-manager-5756559977-t8sfj\" (UID: \"44dd7d23-a748-44a0-8cbb-15a010bef950\") " pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.656316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.656295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:45.792199 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:45.792182    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"]
Jul 14 02:07:45.794935 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:07:45.794909    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dd7d23_a748_44a0_8cbb_15a010bef950.slice/crio-f8773def46d17305416b21d8a732382c5e9505d573340483c08ec4490b8c9dc9 WatchSource:0}: Error finding container f8773def46d17305416b21d8a732382c5e9505d573340483c08ec4490b8c9dc9: Status 404 returned error can't find the container with id f8773def46d17305416b21d8a732382c5e9505d573340483c08ec4490b8c9dc9
Jul 14 02:07:46.326918 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:46.326885    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj" event={"ID":"44dd7d23-a748-44a0-8cbb-15a010bef950","Type":"ContainerStarted","Data":"f8773def46d17305416b21d8a732382c5e9505d573340483c08ec4490b8c9dc9"}
Jul 14 02:07:49.339353 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:49.339316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj" event={"ID":"44dd7d23-a748-44a0-8cbb-15a010bef950","Type":"ContainerStarted","Data":"c0b9b46e115062891b877da32dfb1faaf47a36893ae632e399d3e52307467e38"}
Jul 14 02:07:50.344289 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:50.344200    2584 generic.go:356] "Generic (PLEG): container finished" podID="44dd7d23-a748-44a0-8cbb-15a010bef950" containerID="c0b9b46e115062891b877da32dfb1faaf47a36893ae632e399d3e52307467e38" exitCode=0
Jul 14 02:07:50.344630 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:50.344284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj" event={"ID":"44dd7d23-a748-44a0-8cbb-15a010bef950","Type":"ContainerDied","Data":"c0b9b46e115062891b877da32dfb1faaf47a36893ae632e399d3e52307467e38"}
Jul 14 02:07:51.349819 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:51.349787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj" event={"ID":"44dd7d23-a748-44a0-8cbb-15a010bef950","Type":"ContainerStarted","Data":"299899d5a48809b1225c8ef298796cdbc3f3d09e51edee3c4cfdedf3c72d8b87"}
Jul 14 02:07:51.350222 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:51.349911    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:07:51.371326 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:07:51.371283    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj" podStartSLOduration=1.897233816 podStartE2EDuration="6.371270006s" totalImagesPullingTime="3.381972758s" totalInitContainerRuntime="1.092063432s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:07:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:07:51.369695461 +0000 UTC m=+365.042570262" watchObservedRunningTime="2026-07-14 02:07:51.371270006 +0000 UTC m=+365.044144805"
Jul 14 02:08:02.355557 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:02.355520    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5756559977-t8sfj"
Jul 14 02:08:15.321952 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.321916    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd74468c-czkql"]
Jul 14 02:08:15.325260 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.325238    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.327769 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.327742    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 02:08:15.327769 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.327763    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 02:08:15.327941 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.327762    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hmpcv\""
Jul 14 02:08:15.328048 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.328034    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 02:08:15.328906 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.328891    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 02:08:15.336465 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.336444    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd74468c-czkql"]
Jul 14 02:08:15.430660 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.430622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc457de-e041-4eef-b19a-6ac4a064e66c-tmp\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.430895 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.430674    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjc26\" (UniqueName: \"kubernetes.io/projected/dfc457de-e041-4eef-b19a-6ac4a064e66c-kube-api-access-xjc26\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.430895 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.430714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc457de-e041-4eef-b19a-6ac4a064e66c-tls-certs\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.487605 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.487572    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c6bxl"]
Jul 14 02:08:15.491098 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.491073    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.493809 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.493791    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 14 02:08:15.494521 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.494504    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-84fnf\""
Jul 14 02:08:15.500082 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.500060    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c6bxl"]
Jul 14 02:08:15.531862 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.531835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjc26\" (UniqueName: \"kubernetes.io/projected/dfc457de-e041-4eef-b19a-6ac4a064e66c-kube-api-access-xjc26\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.531992 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.531870    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfxm\" (UniqueName: \"kubernetes.io/projected/3c950f70-bedc-4b2c-9944-8e345814dabe-kube-api-access-tcfxm\") pod \"odh-model-controller-7998c67dc4-c6bxl\" (UID: \"3c950f70-bedc-4b2c-9944-8e345814dabe\") " pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.531992 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.531969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c950f70-bedc-4b2c-9944-8e345814dabe-cert\") pod \"odh-model-controller-7998c67dc4-c6bxl\" (UID: \"3c950f70-bedc-4b2c-9944-8e345814dabe\") " pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.532091 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.532012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc457de-e041-4eef-b19a-6ac4a064e66c-tls-certs\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.532091 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.532044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc457de-e041-4eef-b19a-6ac4a064e66c-tmp\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.534386 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.534364    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc457de-e041-4eef-b19a-6ac4a064e66c-tmp\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.534620 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.534598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc457de-e041-4eef-b19a-6ac4a064e66c-tls-certs\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.542358 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.542334    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjc26\" (UniqueName: \"kubernetes.io/projected/dfc457de-e041-4eef-b19a-6ac4a064e66c-kube-api-access-xjc26\") pod \"kube-auth-proxy-69dd74468c-czkql\" (UID: \"dfc457de-e041-4eef-b19a-6ac4a064e66c\") " pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.632593 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.632562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfxm\" (UniqueName: \"kubernetes.io/projected/3c950f70-bedc-4b2c-9944-8e345814dabe-kube-api-access-tcfxm\") pod \"odh-model-controller-7998c67dc4-c6bxl\" (UID: \"3c950f70-bedc-4b2c-9944-8e345814dabe\") " pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.632801 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.632616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c950f70-bedc-4b2c-9944-8e345814dabe-cert\") pod \"odh-model-controller-7998c67dc4-c6bxl\" (UID: \"3c950f70-bedc-4b2c-9944-8e345814dabe\") " pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.634738 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.634705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql"
Jul 14 02:08:15.635111 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.635089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c950f70-bedc-4b2c-9944-8e345814dabe-cert\") pod \"odh-model-controller-7998c67dc4-c6bxl\" (UID: \"3c950f70-bedc-4b2c-9944-8e345814dabe\") " pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.642304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.642283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfxm\" (UniqueName: \"kubernetes.io/projected/3c950f70-bedc-4b2c-9944-8e345814dabe-kube-api-access-tcfxm\") pod \"odh-model-controller-7998c67dc4-c6bxl\" (UID: \"3c950f70-bedc-4b2c-9944-8e345814dabe\") " pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.766359 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.766331    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd74468c-czkql"]
Jul 14 02:08:15.768946 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:08:15.768916    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc457de_e041_4eef_b19a_6ac4a064e66c.slice/crio-cb01ea3fe7ab01199539a34e91caecfe1b940d703255a4e96593c5a3d7beec7d WatchSource:0}: Error finding container cb01ea3fe7ab01199539a34e91caecfe1b940d703255a4e96593c5a3d7beec7d: Status 404 returned error can't find the container with id cb01ea3fe7ab01199539a34e91caecfe1b940d703255a4e96593c5a3d7beec7d
Jul 14 02:08:15.801618 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.801591    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:15.938079 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:15.938046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c6bxl"]
Jul 14 02:08:15.940701 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:08:15.940675    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c950f70_bedc_4b2c_9944_8e345814dabe.slice/crio-26a16d0d05fedd6048dc45b5d8524caff37aef9257c3c464981f6ae04d20ba72 WatchSource:0}: Error finding container 26a16d0d05fedd6048dc45b5d8524caff37aef9257c3c464981f6ae04d20ba72: Status 404 returned error can't find the container with id 26a16d0d05fedd6048dc45b5d8524caff37aef9257c3c464981f6ae04d20ba72
Jul 14 02:08:16.257240 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.257161    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pjbq5"]
Jul 14 02:08:16.261119 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.261099    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.263636 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.263608    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 14 02:08:16.263725 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.263671    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hbhbx\""
Jul 14 02:08:16.270630 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.270610    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pjbq5"]
Jul 14 02:08:16.339438 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.339403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a-tls-certs\") pod \"model-serving-api-54d89bd5fc-pjbq5\" (UID: \"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.339795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.339445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72zm6\" (UniqueName: \"kubernetes.io/projected/8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a-kube-api-access-72zm6\") pod \"model-serving-api-54d89bd5fc-pjbq5\" (UID: \"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.440488 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.440320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a-tls-certs\") pod \"model-serving-api-54d89bd5fc-pjbq5\" (UID: \"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.440488 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.440371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72zm6\" (UniqueName: \"kubernetes.io/projected/8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a-kube-api-access-72zm6\") pod \"model-serving-api-54d89bd5fc-pjbq5\" (UID: \"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.443077 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.443044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a-tls-certs\") pod \"model-serving-api-54d89bd5fc-pjbq5\" (UID: \"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.444383 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.444349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl" event={"ID":"3c950f70-bedc-4b2c-9944-8e345814dabe","Type":"ContainerStarted","Data":"26a16d0d05fedd6048dc45b5d8524caff37aef9257c3c464981f6ae04d20ba72"}
Jul 14 02:08:16.445783 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.445749    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql" event={"ID":"dfc457de-e041-4eef-b19a-6ac4a064e66c","Type":"ContainerStarted","Data":"cb01ea3fe7ab01199539a34e91caecfe1b940d703255a4e96593c5a3d7beec7d"}
Jul 14 02:08:16.450326 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.450297    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72zm6\" (UniqueName: \"kubernetes.io/projected/8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a-kube-api-access-72zm6\") pod \"model-serving-api-54d89bd5fc-pjbq5\" (UID: \"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a\") " pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.454749 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.454720    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"]
Jul 14 02:08:16.457946 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.457928    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.460741 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.460722    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 14 02:08:16.460741 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.460731    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 02:08:16.460910 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.460728    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gq6pm\""
Jul 14 02:08:16.460910 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.460777    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 14 02:08:16.461161 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.461122    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 14 02:08:16.471942 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.471920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"]
Jul 14 02:08:16.541484 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d906f811-dcf9-4cc5-907a-1865e21121b5-local-certs\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.541484 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-token\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.541484 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.541484 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.541791 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541535    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj54j\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-kube-api-access-fj54j\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.541791 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-cacerts\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.541791 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.541632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.571342 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.571312    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:16.642886 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.642855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d906f811-dcf9-4cc5-907a-1865e21121b5-local-certs\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.643012 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.642909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-token\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.643012 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.642951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.643012 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.642979    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.643196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.643013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj54j\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-kube-api-access-fj54j\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.643196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.643038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-cacerts\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.643196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.643078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.644398 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.644370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.646067 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.646020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d906f811-dcf9-4cc5-907a-1865e21121b5-local-certs\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.646239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.646221    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.646317 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.646297    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-cacerts\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.647378 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.647359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.651992 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.651940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj54j\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-kube-api-access-fj54j\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.651992 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.651956    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-token\") pod \"istiod-openshift-gateway-7574459656-zghvk\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.718150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.717534    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pjbq5"]
Jul 14 02:08:16.721247 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:08:16.721123    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8232901a_bbf1_4110_b0f3_a7c2fc8c7a3a.slice/crio-dc3a29fc38a1b3b4c6b928238497134464b7c5ed7d06d291d35be6af890a1452 WatchSource:0}: Error finding container dc3a29fc38a1b3b4c6b928238497134464b7c5ed7d06d291d35be6af890a1452: Status 404 returned error can't find the container with id dc3a29fc38a1b3b4c6b928238497134464b7c5ed7d06d291d35be6af890a1452
Jul 14 02:08:16.768647 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.768285    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:16.952170 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:16.952118    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"]
Jul 14 02:08:16.956005 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:08:16.955761    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd906f811_dcf9_4cc5_907a_1865e21121b5.slice/crio-922992ac0a5e85f9e0511f5db57e2ef9721f1e2385c52ad44508bb835c1e10d2 WatchSource:0}: Error finding container 922992ac0a5e85f9e0511f5db57e2ef9721f1e2385c52ad44508bb835c1e10d2: Status 404 returned error can't find the container with id 922992ac0a5e85f9e0511f5db57e2ef9721f1e2385c52ad44508bb835c1e10d2
Jul 14 02:08:17.456824 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:17.456783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" event={"ID":"d906f811-dcf9-4cc5-907a-1865e21121b5","Type":"ContainerStarted","Data":"922992ac0a5e85f9e0511f5db57e2ef9721f1e2385c52ad44508bb835c1e10d2"}
Jul 14 02:08:17.458472 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:17.458449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5" event={"ID":"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a","Type":"ContainerStarted","Data":"dc3a29fc38a1b3b4c6b928238497134464b7c5ed7d06d291d35be6af890a1452"}
Jul 14 02:08:21.460635 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:21.460591    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 02:08:21.461012 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:21.460663    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 02:08:22.483258 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.482973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql" event={"ID":"dfc457de-e041-4eef-b19a-6ac4a064e66c","Type":"ContainerStarted","Data":"297f16546bc8472f005568f3cd65ba57c912716b2e2ac1f55c22d104d2e27d4a"}
Jul 14 02:08:22.484745 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.484713    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5" event={"ID":"8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a","Type":"ContainerStarted","Data":"1c65d36b9aa75a409dda9e777beefb3ab7667382d8b33ac4c0fc06fcc89a6fa0"}
Jul 14 02:08:22.484895 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.484805    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:08:22.486324 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.486262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl" event={"ID":"3c950f70-bedc-4b2c-9944-8e345814dabe","Type":"ContainerStarted","Data":"b12207fdaa48d347a43da570adb59ee3f59c332cc0adbbd5c7d222800231661b"}
Jul 14 02:08:22.486464 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.486335    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:22.487972 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.487929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" event={"ID":"d906f811-dcf9-4cc5-907a-1865e21121b5","Type":"ContainerStarted","Data":"8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f"}
Jul 14 02:08:22.488162 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.488147    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:22.489844 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.489822    2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zghvk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 02:08:22.489913 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.489878    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" podUID="d906f811-dcf9-4cc5-907a-1865e21121b5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 02:08:22.500609 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.500560    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69dd74468c-czkql" podStartSLOduration=1.819346612 podStartE2EDuration="7.500545687s" totalImagesPullingTime="5.681199075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:08:22.49969453 +0000 UTC m=+396.172569321" watchObservedRunningTime="2026-07-14 02:08:22.500545687 +0000 UTC m=+396.173420489"
Jul 14 02:08:22.542452 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.542402    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl" podStartSLOduration=2.023093668 podStartE2EDuration="7.542387173s" totalImagesPullingTime="5.519293505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:08:22.517846543 +0000 UTC m=+396.190721345" watchObservedRunningTime="2026-07-14 02:08:22.542387173 +0000 UTC m=+396.215261972"
Jul 14 02:08:22.542609 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.542499    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5" podStartSLOduration=1.8145788619999998 podStartE2EDuration="6.542490106s" totalImagesPullingTime="4.727911244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:08:22.541888821 +0000 UTC m=+396.214763613" watchObservedRunningTime="2026-07-14 02:08:22.542490106 +0000 UTC m=+396.215364907"
Jul 14 02:08:22.563937 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:22.563893    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" podStartSLOduration=2.062040672 podStartE2EDuration="6.563876106s" totalImagesPullingTime="4.501835434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:08:22.56269065 +0000 UTC m=+396.235565452" watchObservedRunningTime="2026-07-14 02:08:22.563876106 +0000 UTC m=+396.236750906"
Jul 14 02:08:23.492773 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:23.492746    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:24.762782 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.762750    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"]
Jul 14 02:08:24.773005 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.772965    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.785063 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.785040    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"]
Jul 14 02:08:24.816200 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0730b6-12a1-4d73-a6ae-77e973f8c011-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.816325 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816220    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsbh\" (UniqueName: \"kubernetes.io/projected/aa0730b6-12a1-4d73-a6ae-77e973f8c011-kube-api-access-phsbh\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.816325 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816286    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.816432 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.816432 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.816432 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.816542 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.816441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917397 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsbh\" (UniqueName: \"kubernetes.io/projected/aa0730b6-12a1-4d73-a6ae-77e973f8c011-kube-api-access-phsbh\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917397 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917397    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917775 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.917775 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.917662    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0730b6-12a1-4d73-a6ae-77e973f8c011-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.918300 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.918274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.919842 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.919815    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.919945 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.919865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.920029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.920007    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0730b6-12a1-4d73-a6ae-77e973f8c011-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.920121 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.920102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.925245 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.925223    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsbh\" (UniqueName: \"kubernetes.io/projected/aa0730b6-12a1-4d73-a6ae-77e973f8c011-kube-api-access-phsbh\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:24.925367 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:24.925350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0730b6-12a1-4d73-a6ae-77e973f8c011-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-d5grx\" (UID: \"aa0730b6-12a1-4d73-a6ae-77e973f8c011\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:25.081932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.081850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:25.212865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.212841    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"]
Jul 14 02:08:25.214799 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:08:25.214769    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0730b6_12a1_4d73_a6ae_77e973f8c011.slice/crio-e69ee1452c65d83ba602e61368c21c9328a11e2f1fab42116ad198b1b2e08690 WatchSource:0}: Error finding container e69ee1452c65d83ba602e61368c21c9328a11e2f1fab42116ad198b1b2e08690: Status 404 returned error can't find the container with id e69ee1452c65d83ba602e61368c21c9328a11e2f1fab42116ad198b1b2e08690
Jul 14 02:08:25.217684 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.217651    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 02:08:25.217778 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.217710    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 02:08:25.502988 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.502952    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx" event={"ID":"aa0730b6-12a1-4d73-a6ae-77e973f8c011","Type":"ContainerStarted","Data":"90619c6701030eade4fc69b41ab7bc5867f9f291e462e29bc2d05b62a8bbcfca"}
Jul 14 02:08:25.503172 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.503009    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx" event={"ID":"aa0730b6-12a1-4d73-a6ae-77e973f8c011","Type":"ContainerStarted","Data":"e69ee1452c65d83ba602e61368c21c9328a11e2f1fab42116ad198b1b2e08690"}
Jul 14 02:08:25.503172 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.503107    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:25.542652 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:25.542603    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx" podStartSLOduration=1.5425876889999999 podStartE2EDuration="1.542587689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:08:25.539830855 +0000 UTC m=+399.212705656" watchObservedRunningTime="2026-07-14 02:08:25.542587689 +0000 UTC m=+399.215462489"
Jul 14 02:08:26.508946 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.508920    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d5grx"
Jul 14 02:08:26.605049 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.605018    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"]
Jul 14 02:08:26.605234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.605211    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" podUID="d906f811-dcf9-4cc5-907a-1865e21121b5" containerName="discovery" containerID="cri-o://8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f" gracePeriod=30
Jul 14 02:08:26.846308 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.846287    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:26.937177 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937149    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj54j\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-kube-api-access-fj54j\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d906f811-dcf9-4cc5-907a-1865e21121b5-local-certs\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937215    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-dns-cert\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937266    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-token\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937282    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-kubeconfig\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937319 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937314    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-ca-configmap\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937575 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937339    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-cacerts\") pod \"d906f811-dcf9-4cc5-907a-1865e21121b5\" (UID: \"d906f811-dcf9-4cc5-907a-1865e21121b5\") "
Jul 14 02:08:26.937809 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.937756    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:08:26.939782 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.939752    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-cacerts" (OuterVolumeSpecName: "cacerts") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:08:26.939874 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.939752    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:08:26.939932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.939883    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-kube-api-access-fj54j" (OuterVolumeSpecName: "kube-api-access-fj54j") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "kube-api-access-fj54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:08:26.939932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.939888    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:08:26.939932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.939905    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:08:26.939932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:26.939900    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d906f811-dcf9-4cc5-907a-1865e21121b5-local-certs" (OuterVolumeSpecName: "local-certs") pod "d906f811-dcf9-4cc5-907a-1865e21121b5" (UID: "d906f811-dcf9-4cc5-907a-1865e21121b5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:08:27.038194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038098    2584 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d906f811-dcf9-4cc5-907a-1865e21121b5-local-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.038194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038126    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-dns-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.038194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038153    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-token\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.038194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038164    2584 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-kubeconfig\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.038194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038177    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d906f811-dcf9-4cc5-907a-1865e21121b5-istio-csr-ca-configmap\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.038194 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038190    2584 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d906f811-dcf9-4cc5-907a-1865e21121b5-cacerts\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.038458 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.038202    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj54j\" (UniqueName: \"kubernetes.io/projected/d906f811-dcf9-4cc5-907a-1865e21121b5-kube-api-access-fj54j\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:08:27.512541 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.512508    2584 generic.go:356] "Generic (PLEG): container finished" podID="d906f811-dcf9-4cc5-907a-1865e21121b5" containerID="8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f" exitCode=0
Jul 14 02:08:27.512924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.512561    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"
Jul 14 02:08:27.512924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.512594    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" event={"ID":"d906f811-dcf9-4cc5-907a-1865e21121b5","Type":"ContainerDied","Data":"8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f"}
Jul 14 02:08:27.512924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.512648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zghvk" event={"ID":"d906f811-dcf9-4cc5-907a-1865e21121b5","Type":"ContainerDied","Data":"922992ac0a5e85f9e0511f5db57e2ef9721f1e2385c52ad44508bb835c1e10d2"}
Jul 14 02:08:27.512924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.512674    2584 scope.go:117] "RemoveContainer" containerID="8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f"
Jul 14 02:08:27.522054 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.522038    2584 scope.go:117] "RemoveContainer" containerID="8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f"
Jul 14 02:08:27.522386 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:08:27.522367    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f\": container with ID starting with 8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f not found: ID does not exist" containerID="8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f"
Jul 14 02:08:27.522437 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.522394    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f"} err="failed to get container status \"8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f\": rpc error: code = NotFound desc = could not find container \"8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f\": container with ID starting with 8f4df5fa70b98c576a85c79de6459e031af7b309e32bc009b9ec710cf49cd31f not found: ID does not exist"
Jul 14 02:08:27.560106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.560080    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"]
Jul 14 02:08:27.565322 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:27.565300    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zghvk"]
Jul 14 02:08:28.890867 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:28.890831    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d906f811-dcf9-4cc5-907a-1865e21121b5" path="/var/lib/kubelet/pods/d906f811-dcf9-4cc5-907a-1865e21121b5/volumes"
Jul 14 02:08:33.494990 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:33.494958    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-c6bxl"
Jul 14 02:08:33.497550 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:08:33.497529    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pjbq5"
Jul 14 02:09:06.082262 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.082224    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tgckv"]
Jul 14 02:09:06.082711 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.082495    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d906f811-dcf9-4cc5-907a-1865e21121b5" containerName="discovery"
Jul 14 02:09:06.082711 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.082506    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d906f811-dcf9-4cc5-907a-1865e21121b5" containerName="discovery"
Jul 14 02:09:06.082711 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.082556    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d906f811-dcf9-4cc5-907a-1865e21121b5" containerName="discovery"
Jul 14 02:09:06.089516 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.089493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:06.092039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.092013    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 02:09:06.093059 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.093027    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 02:09:06.093230 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.093050    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-l8z8b\""
Jul 14 02:09:06.093720 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.093692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tgckv"]
Jul 14 02:09:06.131382 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.131347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zlv\" (UniqueName: \"kubernetes.io/projected/82bf3890-e4c7-4281-9c57-655864e60cc2-kube-api-access-j6zlv\") pod \"kuadrant-operator-catalog-tgckv\" (UID: \"82bf3890-e4c7-4281-9c57-655864e60cc2\") " pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:06.232470 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.232440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zlv\" (UniqueName: \"kubernetes.io/projected/82bf3890-e4c7-4281-9c57-655864e60cc2-kube-api-access-j6zlv\") pod \"kuadrant-operator-catalog-tgckv\" (UID: \"82bf3890-e4c7-4281-9c57-655864e60cc2\") " pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:06.240691 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.240669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zlv\" (UniqueName: \"kubernetes.io/projected/82bf3890-e4c7-4281-9c57-655864e60cc2-kube-api-access-j6zlv\") pod \"kuadrant-operator-catalog-tgckv\" (UID: \"82bf3890-e4c7-4281-9c57-655864e60cc2\") " pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:06.400911 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.400882    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:06.456001 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.455973    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tgckv"]
Jul 14 02:09:06.523648 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.523626    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tgckv"]
Jul 14 02:09:06.525321 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:09:06.525293    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bf3890_e4c7_4281_9c57_655864e60cc2.slice/crio-8ca9378eaec92c479e051b616203f675d31e3f2b2004f09c0a753b350282f20b WatchSource:0}: Error finding container 8ca9378eaec92c479e051b616203f675d31e3f2b2004f09c0a753b350282f20b: Status 404 returned error can't find the container with id 8ca9378eaec92c479e051b616203f675d31e3f2b2004f09c0a753b350282f20b
Jul 14 02:09:06.662166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.662078    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4gmpt"]
Jul 14 02:09:06.665296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.665271    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tgckv" event={"ID":"82bf3890-e4c7-4281-9c57-655864e60cc2","Type":"ContainerStarted","Data":"8ca9378eaec92c479e051b616203f675d31e3f2b2004f09c0a753b350282f20b"}
Jul 14 02:09:06.665426 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.665349    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:06.672927 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.672902    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4gmpt"]
Jul 14 02:09:06.735803 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.735774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d828\" (UniqueName: \"kubernetes.io/projected/93a4f01a-afd0-4db5-904c-cdecc5fd29d2-kube-api-access-6d828\") pod \"kuadrant-operator-catalog-4gmpt\" (UID: \"93a4f01a-afd0-4db5-904c-cdecc5fd29d2\") " pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:06.836570 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.836536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d828\" (UniqueName: \"kubernetes.io/projected/93a4f01a-afd0-4db5-904c-cdecc5fd29d2-kube-api-access-6d828\") pod \"kuadrant-operator-catalog-4gmpt\" (UID: \"93a4f01a-afd0-4db5-904c-cdecc5fd29d2\") " pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:06.844917 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.844887    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d828\" (UniqueName: \"kubernetes.io/projected/93a4f01a-afd0-4db5-904c-cdecc5fd29d2-kube-api-access-6d828\") pod \"kuadrant-operator-catalog-4gmpt\" (UID: \"93a4f01a-afd0-4db5-904c-cdecc5fd29d2\") " pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:06.974736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:06.974639    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:07.103946 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:07.103922    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4gmpt"]
Jul 14 02:09:07.105852 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:09:07.105830    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a4f01a_afd0_4db5_904c_cdecc5fd29d2.slice/crio-667b74b013ed1d3b7a0a13cf75aeeef17e9adb5126b03b8080e50f7a786bc16d WatchSource:0}: Error finding container 667b74b013ed1d3b7a0a13cf75aeeef17e9adb5126b03b8080e50f7a786bc16d: Status 404 returned error can't find the container with id 667b74b013ed1d3b7a0a13cf75aeeef17e9adb5126b03b8080e50f7a786bc16d
Jul 14 02:09:07.671804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:07.671750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt" event={"ID":"93a4f01a-afd0-4db5-904c-cdecc5fd29d2","Type":"ContainerStarted","Data":"667b74b013ed1d3b7a0a13cf75aeeef17e9adb5126b03b8080e50f7a786bc16d"}
Jul 14 02:09:08.700144 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:09:08.700093    2584 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:dd64bf2dd177757451a98fcdc999a339c35dee5d9872d8f4dc69c8f3c4dd0112: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" image="quay.io/kuadrant/kuadrant-operator-catalog:v1.4.2"
Jul 14 02:09:08.700480 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:09:08.700231    2584 kuberuntime_manager.go:1459] "Unhandled Error" err="container registry-server start failed in pod kuadrant-operator-catalog-4gmpt_kuadrant-system(93a4f01a-afd0-4db5-904c-cdecc5fd29d2): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:dd64bf2dd177757451a98fcdc999a339c35dee5d9872d8f4dc69c8f3c4dd0112: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 14 02:09:08.700480 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:09:08.700265    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"registry-server\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:dd64bf2dd177757451a98fcdc999a339c35dee5d9872d8f4dc69c8f3c4dd0112: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt" podUID="93a4f01a-afd0-4db5-904c-cdecc5fd29d2"
Jul 14 02:09:09.682004 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:09.681963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tgckv" event={"ID":"82bf3890-e4c7-4281-9c57-655864e60cc2","Type":"ContainerStarted","Data":"9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9"}
Jul 14 02:09:09.682191 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:09.682010    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tgckv" podUID="82bf3890-e4c7-4281-9c57-655864e60cc2" containerName="registry-server" containerID="cri-o://9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9" gracePeriod=2
Jul 14 02:09:09.682635 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:09:09.682538    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"registry-server\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/kuadrant/kuadrant-operator-catalog:v1.4.2\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:dd64bf2dd177757451a98fcdc999a339c35dee5d9872d8f4dc69c8f3c4dd0112: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt" podUID="93a4f01a-afd0-4db5-904c-cdecc5fd29d2"
Jul 14 02:09:09.714242 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:09.714194    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tgckv" podStartSLOduration=1.507877234 podStartE2EDuration="3.714179923s" totalImagesPullingTime="2.206302689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:09:09.713082841 +0000 UTC m=+443.385957641" watchObservedRunningTime="2026-07-14 02:09:09.714179923 +0000 UTC m=+443.387054744"
Jul 14 02:09:09.919278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:09.919255    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:09.962835 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:09.962771    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6zlv\" (UniqueName: \"kubernetes.io/projected/82bf3890-e4c7-4281-9c57-655864e60cc2-kube-api-access-j6zlv\") pod \"82bf3890-e4c7-4281-9c57-655864e60cc2\" (UID: \"82bf3890-e4c7-4281-9c57-655864e60cc2\") "
Jul 14 02:09:09.964948 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:09.964925    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bf3890-e4c7-4281-9c57-655864e60cc2-kube-api-access-j6zlv" (OuterVolumeSpecName: "kube-api-access-j6zlv") pod "82bf3890-e4c7-4281-9c57-655864e60cc2" (UID: "82bf3890-e4c7-4281-9c57-655864e60cc2"). InnerVolumeSpecName "kube-api-access-j6zlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:09:10.064074 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.064033    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zlv\" (UniqueName: \"kubernetes.io/projected/82bf3890-e4c7-4281-9c57-655864e60cc2-kube-api-access-j6zlv\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:09:10.686807 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.686769    2584 generic.go:356] "Generic (PLEG): container finished" podID="82bf3890-e4c7-4281-9c57-655864e60cc2" containerID="9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9" exitCode=0
Jul 14 02:09:10.686979 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.686814    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tgckv"
Jul 14 02:09:10.686979 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.686853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tgckv" event={"ID":"82bf3890-e4c7-4281-9c57-655864e60cc2","Type":"ContainerDied","Data":"9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9"}
Jul 14 02:09:10.686979 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.686906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tgckv" event={"ID":"82bf3890-e4c7-4281-9c57-655864e60cc2","Type":"ContainerDied","Data":"8ca9378eaec92c479e051b616203f675d31e3f2b2004f09c0a753b350282f20b"}
Jul 14 02:09:10.686979 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.686930    2584 scope.go:117] "RemoveContainer" containerID="9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9"
Jul 14 02:09:10.695549 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.695526    2584 scope.go:117] "RemoveContainer" containerID="9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9"
Jul 14 02:09:10.695785 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:09:10.695769    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9\": container with ID starting with 9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9 not found: ID does not exist" containerID="9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9"
Jul 14 02:09:10.695828 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.695793    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9"} err="failed to get container status \"9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9\": rpc error: code = NotFound desc = could not find container \"9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9\": container with ID starting with 9c1c18fc056f59b93d4cefa173671ebed6ab85050384d2289728c6a4a613a0f9 not found: ID does not exist"
Jul 14 02:09:10.706545 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.706523    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tgckv"]
Jul 14 02:09:10.710410 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.710387    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tgckv"]
Jul 14 02:09:10.892537 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:10.892508    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82bf3890-e4c7-4281-9c57-655864e60cc2" path="/var/lib/kubelet/pods/82bf3890-e4c7-4281-9c57-655864e60cc2/volumes"
Jul 14 02:09:24.742833 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:24.742798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt" event={"ID":"93a4f01a-afd0-4db5-904c-cdecc5fd29d2","Type":"ContainerStarted","Data":"6a0b63e0f7dd22fab7493f3e177219f936c37078e3cd2c790c878d43811b527b"}
Jul 14 02:09:24.758693 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:24.758643    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt" podStartSLOduration=1.6047775770000001 podStartE2EDuration="18.758629236s" totalImagesPullingTime="17.153851659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-14 02:09:24.758023795 +0000 UTC m=+458.430898611" watchObservedRunningTime="2026-07-14 02:09:24.758629236 +0000 UTC m=+458.431504035"
Jul 14 02:09:26.975804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:26.975774    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:26.976212 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:26.975830    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:26.998335 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:26.998310    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:37.779252 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:37.779224    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4gmpt"
Jul 14 02:09:54.970875 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.970846    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"]
Jul 14 02:09:54.971414 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.971103    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bf3890-e4c7-4281-9c57-655864e60cc2" containerName="registry-server"
Jul 14 02:09:54.971414 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.971115    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bf3890-e4c7-4281-9c57-655864e60cc2" containerName="registry-server"
Jul 14 02:09:54.971414 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.971199    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bf3890-e4c7-4281-9c57-655864e60cc2" containerName="registry-server"
Jul 14 02:09:54.975028 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.975011    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:09:54.978044 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.978023    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pchft\""
Jul 14 02:09:54.994370 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:54.994347    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"]
Jul 14 02:09:55.098342 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:55.098305    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2vt\" (UniqueName: \"kubernetes.io/projected/0125c9ce-c8b2-4bd9-8d26-602e0f592a05-kube-api-access-hs2vt\") pod \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" (UID: \"0125c9ce-c8b2-4bd9-8d26-602e0f592a05\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:09:55.199430 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:55.199402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2vt\" (UniqueName: \"kubernetes.io/projected/0125c9ce-c8b2-4bd9-8d26-602e0f592a05-kube-api-access-hs2vt\") pod \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" (UID: \"0125c9ce-c8b2-4bd9-8d26-602e0f592a05\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:09:55.210665 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:55.210635    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2vt\" (UniqueName: \"kubernetes.io/projected/0125c9ce-c8b2-4bd9-8d26-602e0f592a05-kube-api-access-hs2vt\") pod \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" (UID: \"0125c9ce-c8b2-4bd9-8d26-602e0f592a05\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:09:55.284765 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:55.284692    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:09:55.414525 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:55.414500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"]
Jul 14 02:09:55.416877 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:09:55.416852    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0125c9ce_c8b2_4bd9_8d26_602e0f592a05.slice/crio-b2a963de040b28bbe29d6b8fd38ed2012496cc43f8b7e3fa4391f011285639c1 WatchSource:0}: Error finding container b2a963de040b28bbe29d6b8fd38ed2012496cc43f8b7e3fa4391f011285639c1: Status 404 returned error can't find the container with id b2a963de040b28bbe29d6b8fd38ed2012496cc43f8b7e3fa4391f011285639c1
Jul 14 02:09:55.867279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:55.867245    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm" event={"ID":"0125c9ce-c8b2-4bd9-8d26-602e0f592a05","Type":"ContainerStarted","Data":"b2a963de040b28bbe29d6b8fd38ed2012496cc43f8b7e3fa4391f011285639c1"}
Jul 14 02:09:57.877368 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:57.877335    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm" event={"ID":"0125c9ce-c8b2-4bd9-8d26-602e0f592a05","Type":"ContainerStarted","Data":"80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22"}
Jul 14 02:09:57.877742 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:57.877409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:09:57.893076 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:57.893027    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm" podStartSLOduration=2.120739626 podStartE2EDuration="3.893013364s" totalImagesPullingTime="1.772273738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:09:57.892031303 +0000 UTC m=+491.564906104" watchObservedRunningTime="2026-07-14 02:09:57.893013364 +0000 UTC m=+491.565888221"
Jul 14 02:09:58.988702 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:58.988674    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"]
Jul 14 02:09:58.991886 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:58.991867    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:09:58.994438 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:58.994419    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jqzvp\""
Jul 14 02:09:59.001883 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.001865    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"]
Jul 14 02:09:59.132350 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.132316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqmg\" (UniqueName: \"kubernetes.io/projected/21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e-kube-api-access-hvqmg\") pod \"authorino-operator-75cc8d5cb7-g5fmq\" (UID: \"21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:09:59.232900 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.232868    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqmg\" (UniqueName: \"kubernetes.io/projected/21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e-kube-api-access-hvqmg\") pod \"authorino-operator-75cc8d5cb7-g5fmq\" (UID: \"21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:09:59.241230 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.241169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqmg\" (UniqueName: \"kubernetes.io/projected/21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e-kube-api-access-hvqmg\") pod \"authorino-operator-75cc8d5cb7-g5fmq\" (UID: \"21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:09:59.302234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.302201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:09:59.426939 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.426814    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"]
Jul 14 02:09:59.429658 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:09:59.429631    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c0a6f0_bac6_4c3d_bc7a_c464f1e9d41e.slice/crio-95fb7c811783f422818ffc0de439ab195269296d98145c35ad3229926283e4ab WatchSource:0}: Error finding container 95fb7c811783f422818ffc0de439ab195269296d98145c35ad3229926283e4ab: Status 404 returned error can't find the container with id 95fb7c811783f422818ffc0de439ab195269296d98145c35ad3229926283e4ab
Jul 14 02:09:59.885774 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:09:59.885741    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq" event={"ID":"21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e","Type":"ContainerStarted","Data":"95fb7c811783f422818ffc0de439ab195269296d98145c35ad3229926283e4ab"}
Jul 14 02:10:01.903570 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:01.903530    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq" event={"ID":"21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e","Type":"ContainerStarted","Data":"9a78cbb0d6dfff58f7aa039c6ef190c923bc928cfa2a4f02175dd888c773abb3"}
Jul 14 02:10:01.903940 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:01.903671    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:10:01.918702 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:01.918657    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq" podStartSLOduration=1.649052051 podStartE2EDuration="3.918644915s" totalImagesPullingTime="2.269592864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:10:01.917785665 +0000 UTC m=+495.590660466" watchObservedRunningTime="2026-07-14 02:10:01.918644915 +0000 UTC m=+495.591519714"
Jul 14 02:10:08.883623 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:08.883583    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:12.911901 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:12.911874    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g5fmq"
Jul 14 02:10:26.606721 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.606680    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"]
Jul 14 02:10:26.610123 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.610101    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.612546 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.612526    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-st7t7\""
Jul 14 02:10:26.622987 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.622966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"]
Jul 14 02:10:26.739332 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.739296    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8665cd28-a170-4e6a-b54e-62453b0dfd53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.739483 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.739340    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2kq\" (UniqueName: \"kubernetes.io/projected/8665cd28-a170-4e6a-b54e-62453b0dfd53-kube-api-access-kv2kq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.840597 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.840565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2kq\" (UniqueName: \"kubernetes.io/projected/8665cd28-a170-4e6a-b54e-62453b0dfd53-kube-api-access-kv2kq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.840755 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.840656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8665cd28-a170-4e6a-b54e-62453b0dfd53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.841038 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.841019    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8665cd28-a170-4e6a-b54e-62453b0dfd53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.849870 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.849847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2kq\" (UniqueName: \"kubernetes.io/projected/8665cd28-a170-4e6a-b54e-62453b0dfd53-kube-api-access-kv2kq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:26.919982 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:26.919961    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:27.053986 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.053923    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"]
Jul 14 02:10:27.056635 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:10:27.056606    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8665cd28_a170_4e6a_b54e_62453b0dfd53.slice/crio-614104f9dea4621038b3a4683a93848f4c1f84582dd1e9cae063c09fc47df362 WatchSource:0}: Error finding container 614104f9dea4621038b3a4683a93848f4c1f84582dd1e9cae063c09fc47df362: Status 404 returned error can't find the container with id 614104f9dea4621038b3a4683a93848f4c1f84582dd1e9cae063c09fc47df362
Jul 14 02:10:27.238688 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.238606    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"]
Jul 14 02:10:27.245832 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.245805    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"]
Jul 14 02:10:27.260661 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.260635    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"]
Jul 14 02:10:27.265084 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.265062    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.278282 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.278253    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"]
Jul 14 02:10:27.288487 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.288454    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"]
Jul 14 02:10:27.288683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.288659    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" containerName="manager" containerID="cri-o://80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22" gracePeriod=2
Jul 14 02:10:27.299736 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:27.299690    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:27.300244 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.300181    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"]
Jul 14 02:10:27.313741 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.313718    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"]
Jul 14 02:10:27.313994 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.313982    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" containerName="manager"
Jul 14 02:10:27.314037 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.313995    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" containerName="manager"
Jul 14 02:10:27.314070 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.314050    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" containerName="manager"
Jul 14 02:10:27.316936 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.316917    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:27.336850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.336825    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"]
Jul 14 02:10:27.346755 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:27.346727    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:27.445593 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.445559    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jgtjf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.445740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.445597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rrd\" (UniqueName: \"kubernetes.io/projected/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-kube-api-access-g5rrd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jgtjf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.445740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.445687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76qv\" (UniqueName: \"kubernetes.io/projected/9b72815e-a93c-4194-bb2e-ed818d6b36c4-kube-api-access-z76qv\") pod \"limitador-operator-controller-manager-859d7b78bd-bbqp9\" (UID: \"9b72815e-a93c-4194-bb2e-ed818d6b36c4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:27.521158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.521119    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:27.523594 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:27.523551    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:27.547013 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.546981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76qv\" (UniqueName: \"kubernetes.io/projected/9b72815e-a93c-4194-bb2e-ed818d6b36c4-kube-api-access-z76qv\") pod \"limitador-operator-controller-manager-859d7b78bd-bbqp9\" (UID: \"9b72815e-a93c-4194-bb2e-ed818d6b36c4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:27.547166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.547104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jgtjf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.547166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.547158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rrd\" (UniqueName: \"kubernetes.io/projected/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-kube-api-access-g5rrd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jgtjf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.547677 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.547651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jgtjf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.562458 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.562420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76qv\" (UniqueName: \"kubernetes.io/projected/9b72815e-a93c-4194-bb2e-ed818d6b36c4-kube-api-access-z76qv\") pod \"limitador-operator-controller-manager-859d7b78bd-bbqp9\" (UID: \"9b72815e-a93c-4194-bb2e-ed818d6b36c4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:27.562703 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.562680    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rrd\" (UniqueName: \"kubernetes.io/projected/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-kube-api-access-g5rrd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jgtjf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.575202 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.575179    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:27.648557 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.648395    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs2vt\" (UniqueName: \"kubernetes.io/projected/0125c9ce-c8b2-4bd9-8d26-602e0f592a05-kube-api-access-hs2vt\") pod \"0125c9ce-c8b2-4bd9-8d26-602e0f592a05\" (UID: \"0125c9ce-c8b2-4bd9-8d26-602e0f592a05\") "
Jul 14 02:10:27.650839 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.650803    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0125c9ce-c8b2-4bd9-8d26-602e0f592a05-kube-api-access-hs2vt" (OuterVolumeSpecName: "kube-api-access-hs2vt") pod "0125c9ce-c8b2-4bd9-8d26-602e0f592a05" (UID: "0125c9ce-c8b2-4bd9-8d26-602e0f592a05"). InnerVolumeSpecName "kube-api-access-hs2vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:10:27.669535 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.669255    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:27.730813 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.730694    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"]
Jul 14 02:10:27.735558 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:10:27.735531    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2786b7f_e7f4_4b5b_962a_c033fa990bcf.slice/crio-c3236d1f4a7998cc9775f9a2d2a015160a951ad6e9895d58b41224d664ce892a WatchSource:0}: Error finding container c3236d1f4a7998cc9775f9a2d2a015160a951ad6e9895d58b41224d664ce892a: Status 404 returned error can't find the container with id c3236d1f4a7998cc9775f9a2d2a015160a951ad6e9895d58b41224d664ce892a
Jul 14 02:10:27.749854 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.749829    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2vt\" (UniqueName: \"kubernetes.io/projected/0125c9ce-c8b2-4bd9-8d26-602e0f592a05-kube-api-access-hs2vt\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:10:27.827015 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:27.826930    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"]
Jul 14 02:10:27.830266 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:10:27.830236    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b72815e_a93c_4194_bb2e_ed818d6b36c4.slice/crio-9641db6ba9b07fe1ac860c0190887dcb7be2bbd72a124d54f448745a9a2d029f WatchSource:0}: Error finding container 9641db6ba9b07fe1ac860c0190887dcb7be2bbd72a124d54f448745a9a2d029f: Status 404 returned error can't find the container with id 9641db6ba9b07fe1ac860c0190887dcb7be2bbd72a124d54f448745a9a2d029f
Jul 14 02:10:28.011701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.011663    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9" event={"ID":"9b72815e-a93c-4194-bb2e-ed818d6b36c4","Type":"ContainerStarted","Data":"e3960db5a875b16ff01d4df3856237878c06e8c4d28e0397eabb43eca83d0827"}
Jul 14 02:10:28.011877 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.011712    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9" event={"ID":"9b72815e-a93c-4194-bb2e-ed818d6b36c4","Type":"ContainerStarted","Data":"9641db6ba9b07fe1ac860c0190887dcb7be2bbd72a124d54f448745a9a2d029f"}
Jul 14 02:10:28.011877 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.011747    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:28.013985 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.013952    2584 generic.go:356] "Generic (PLEG): container finished" podID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" containerID="80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22" exitCode=0
Jul 14 02:10:28.014104 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.014000    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:28.014104 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.014048    2584 scope.go:117] "RemoveContainer" containerID="80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22"
Jul 14 02:10:28.015451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.015420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf" event={"ID":"f2786b7f-e7f4-4b5b-962a-c033fa990bcf","Type":"ContainerStarted","Data":"c3236d1f4a7998cc9775f9a2d2a015160a951ad6e9895d58b41224d664ce892a"}
Jul 14 02:10:28.022410 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.022395    2584 scope.go:117] "RemoveContainer" containerID="80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22"
Jul 14 02:10:28.022647 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:28.022630    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22\": container with ID starting with 80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22 not found: ID does not exist" containerID="80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22"
Jul 14 02:10:28.022702 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.022657    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22"} err="failed to get container status \"80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22\": rpc error: code = NotFound desc = could not find container \"80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22\": container with ID starting with 80ad9b86fec2d1f45ee2e5545156363c9726055993f2f40dca8ec5ce24228c22 not found: ID does not exist"
Jul 14 02:10:28.035271 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.035212    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9" podStartSLOduration=1.035197108 podStartE2EDuration="1.035197108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:10:28.033735105 +0000 UTC m=+521.706609906" watchObservedRunningTime="2026-07-14 02:10:28.035197108 +0000 UTC m=+521.708071909"
Jul 14 02:10:28.035783 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:28.035758    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:28.037647 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:28.037624    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7mqbm\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7mqbm"
Jul 14 02:10:28.893067 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:28.893027    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0125c9ce-c8b2-4bd9-8d26-602e0f592a05" path="/var/lib/kubelet/pods/0125c9ce-c8b2-4bd9-8d26-602e0f592a05/volumes"
Jul 14 02:10:33.039983 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.039942    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" containerName="manager" containerID="cri-o://6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d" gracePeriod=2
Jul 14 02:10:33.041576 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.041537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf" event={"ID":"f2786b7f-e7f4-4b5b-962a-c033fa990bcf","Type":"ContainerStarted","Data":"1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb"}
Jul 14 02:10:33.041703 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.041667    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:33.061019 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.060970    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf" podStartSLOduration=1.825117866 podStartE2EDuration="6.060945755s" totalImagesPullingTime="4.235827889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:10:33.059554719 +0000 UTC m=+526.732429518" watchObservedRunningTime="2026-07-14 02:10:33.060945755 +0000 UTC m=+526.733820556"
Jul 14 02:10:33.279022 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.279001    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:33.281612 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:33.281583    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:33.295891 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.295847    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv2kq\" (UniqueName: \"kubernetes.io/projected/8665cd28-a170-4e6a-b54e-62453b0dfd53-kube-api-access-kv2kq\") pod \"8665cd28-a170-4e6a-b54e-62453b0dfd53\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") "
Jul 14 02:10:33.295967 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.295910    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8665cd28-a170-4e6a-b54e-62453b0dfd53-extensions-socket-volume\") pod \"8665cd28-a170-4e6a-b54e-62453b0dfd53\" (UID: \"8665cd28-a170-4e6a-b54e-62453b0dfd53\") "
Jul 14 02:10:33.296121 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.296101    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8665cd28-a170-4e6a-b54e-62453b0dfd53-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8665cd28-a170-4e6a-b54e-62453b0dfd53" (UID: "8665cd28-a170-4e6a-b54e-62453b0dfd53"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:10:33.297794 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.297775    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8665cd28-a170-4e6a-b54e-62453b0dfd53-kube-api-access-kv2kq" (OuterVolumeSpecName: "kube-api-access-kv2kq") pod "8665cd28-a170-4e6a-b54e-62453b0dfd53" (UID: "8665cd28-a170-4e6a-b54e-62453b0dfd53"). InnerVolumeSpecName "kube-api-access-kv2kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:10:33.397229 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.397204    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8665cd28-a170-4e6a-b54e-62453b0dfd53-extensions-socket-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:10:33.397229 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:33.397227    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2kq\" (UniqueName: \"kubernetes.io/projected/8665cd28-a170-4e6a-b54e-62453b0dfd53-kube-api-access-kv2kq\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:10:34.046946 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:34.046912    2584 generic.go:356] "Generic (PLEG): container finished" podID="8665cd28-a170-4e6a-b54e-62453b0dfd53" containerID="6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d" exitCode=2
Jul 14 02:10:34.047415 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:34.046960    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:34.047415 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:34.047017    2584 scope.go:117] "RemoveContainer" containerID="6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d"
Jul 14 02:10:34.049333 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:34.049307    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:34.055495 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:34.055479    2584 scope.go:117] "RemoveContainer" containerID="6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d"
Jul 14 02:10:34.055734 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:34.055716    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d\": container with ID starting with 6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d not found: ID does not exist" containerID="6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d"
Jul 14 02:10:34.055781 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:34.055742    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d"} err="failed to get container status \"6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d\": rpc error: code = NotFound desc = could not find container \"6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d\": container with ID starting with 6d18674b6241b779e35d3c3747850ec0f9cae1648f27dc330ca56f526541540d not found: ID does not exist"
Jul 14 02:10:34.057120 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:34.057103    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-64hn5\" is forbidden: User \"system:node:ip-10-0-128-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-94.ec2.internal' and this object" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64hn5"
Jul 14 02:10:34.891327 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:34.891289    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" path="/var/lib/kubelet/pods/8665cd28-a170-4e6a-b54e-62453b0dfd53/volumes"
Jul 14 02:10:39.024418 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:39.024338    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bbqp9"
Jul 14 02:10:44.049464 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:44.049430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:56.773356 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:56.773312    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"]
Jul 14 02:10:56.773826 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:56.773512    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf" podUID="f2786b7f-e7f4-4b5b-962a-c033fa990bcf" containerName="manager" containerID="cri-o://1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb" gracePeriod=10
Jul 14 02:10:57.021737 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.021715    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:57.067010 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.066938    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-extensions-socket-volume\") pod \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") "
Jul 14 02:10:57.067010 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.066972    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5rrd\" (UniqueName: \"kubernetes.io/projected/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-kube-api-access-g5rrd\") pod \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\" (UID: \"f2786b7f-e7f4-4b5b-962a-c033fa990bcf\") "
Jul 14 02:10:57.067327 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.067304    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f2786b7f-e7f4-4b5b-962a-c033fa990bcf" (UID: "f2786b7f-e7f4-4b5b-962a-c033fa990bcf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:10:57.068915 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.068890    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-kube-api-access-g5rrd" (OuterVolumeSpecName: "kube-api-access-g5rrd") pod "f2786b7f-e7f4-4b5b-962a-c033fa990bcf" (UID: "f2786b7f-e7f4-4b5b-962a-c033fa990bcf"). InnerVolumeSpecName "kube-api-access-g5rrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:10:57.143151 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.143098    2584 generic.go:356] "Generic (PLEG): container finished" podID="f2786b7f-e7f4-4b5b-962a-c033fa990bcf" containerID="1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb" exitCode=0
Jul 14 02:10:57.143280 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.143165    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"
Jul 14 02:10:57.143280 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.143189    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf" event={"ID":"f2786b7f-e7f4-4b5b-962a-c033fa990bcf","Type":"ContainerDied","Data":"1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb"}
Jul 14 02:10:57.143280 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.143244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf" event={"ID":"f2786b7f-e7f4-4b5b-962a-c033fa990bcf","Type":"ContainerDied","Data":"c3236d1f4a7998cc9775f9a2d2a015160a951ad6e9895d58b41224d664ce892a"}
Jul 14 02:10:57.143280 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.143267    2584 scope.go:117] "RemoveContainer" containerID="1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb"
Jul 14 02:10:57.151487 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.151471    2584 scope.go:117] "RemoveContainer" containerID="1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb"
Jul 14 02:10:57.151709 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:10:57.151695    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb\": container with ID starting with 1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb not found: ID does not exist" containerID="1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb"
Jul 14 02:10:57.151748 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.151718    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb"} err="failed to get container status \"1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb\": rpc error: code = NotFound desc = could not find container \"1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb\": container with ID starting with 1c19585c0f7f22c0086f2ea0ac84a412e3f0fbafae655567d1fdc76fbfe4eefb not found: ID does not exist"
Jul 14 02:10:57.164574 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.164549    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"]
Jul 14 02:10:57.168278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.168236    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-extensions-socket-volume\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:10:57.168278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.168278    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rrd\" (UniqueName: \"kubernetes.io/projected/f2786b7f-e7f4-4b5b-962a-c033fa990bcf-kube-api-access-g5rrd\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:10:57.168479 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:57.168466    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jgtjf"]
Jul 14 02:10:58.891865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:10:58.891832    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2786b7f-e7f4-4b5b-962a-c033fa990bcf" path="/var/lib/kubelet/pods/f2786b7f-e7f4-4b5b-962a-c033fa990bcf/volumes"
Jul 14 02:11:15.389804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.389766    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:15.390215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.390041    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2786b7f-e7f4-4b5b-962a-c033fa990bcf" containerName="manager"
Jul 14 02:11:15.390215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.390050    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2786b7f-e7f4-4b5b-962a-c033fa990bcf" containerName="manager"
Jul 14 02:11:15.390215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.390072    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" containerName="manager"
Jul 14 02:11:15.390215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.390078    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" containerName="manager"
Jul 14 02:11:15.390215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.390123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2786b7f-e7f4-4b5b-962a-c033fa990bcf" containerName="manager"
Jul 14 02:11:15.390215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.390146    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8665cd28-a170-4e6a-b54e-62453b0dfd53" containerName="manager"
Jul 14 02:11:15.392302 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.392281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.394715 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.394690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 14 02:11:15.394715 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.394703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ggm5q\""
Jul 14 02:11:15.400326 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.400305    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:15.483924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.483893    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:15.505201 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.505166    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhq5h\" (UniqueName: \"kubernetes.io/projected/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-kube-api-access-vhq5h\") pod \"limitador-limitador-64c94fd64f-bkd6x\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.505382 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.505224    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-config-file\") pod \"limitador-limitador-64c94fd64f-bkd6x\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.606328 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.606288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhq5h\" (UniqueName: \"kubernetes.io/projected/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-kube-api-access-vhq5h\") pod \"limitador-limitador-64c94fd64f-bkd6x\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.606493 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.606365    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-config-file\") pod \"limitador-limitador-64c94fd64f-bkd6x\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.606930 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.606907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-config-file\") pod \"limitador-limitador-64c94fd64f-bkd6x\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.613516 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.613490    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhq5h\" (UniqueName: \"kubernetes.io/projected/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-kube-api-access-vhq5h\") pod \"limitador-limitador-64c94fd64f-bkd6x\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.702837 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.702763    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:15.829732 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:15.829708    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:15.832303 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:11:15.832276    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105eb3cf_f9f8_4fd7_9733_5a6a1bcbc328.slice/crio-bf75d574970cdbe5cb89cea254855a01b55568235b989245ed68d974e8cf6814 WatchSource:0}: Error finding container bf75d574970cdbe5cb89cea254855a01b55568235b989245ed68d974e8cf6814: Status 404 returned error can't find the container with id bf75d574970cdbe5cb89cea254855a01b55568235b989245ed68d974e8cf6814
Jul 14 02:11:16.229320 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:16.229285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x" event={"ID":"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328","Type":"ContainerStarted","Data":"bf75d574970cdbe5cb89cea254855a01b55568235b989245ed68d974e8cf6814"}
Jul 14 02:11:19.244213 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:19.244176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x" event={"ID":"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328","Type":"ContainerStarted","Data":"3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad"}
Jul 14 02:11:19.244604 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:19.244248    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:19.261514 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:19.261468    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x" podStartSLOduration=1.314938001 podStartE2EDuration="4.261451326s" totalImagesPullingTime="2.946513325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:11:19.259868943 +0000 UTC m=+572.932743745" watchObservedRunningTime="2026-07-14 02:11:19.261451326 +0000 UTC m=+572.934326127"
Jul 14 02:11:30.249599 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:30.249567    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:32.091697 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.091663    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:32.092108 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.091843    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x" podUID="105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" containerName="limitador" containerID="cri-o://3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad" gracePeriod=30
Jul 14 02:11:32.627353 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.627327    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:32.741923 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.741831    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-config-file\") pod \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") "
Jul 14 02:11:32.741923 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.741893    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhq5h\" (UniqueName: \"kubernetes.io/projected/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-kube-api-access-vhq5h\") pod \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\" (UID: \"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328\") "
Jul 14 02:11:32.742217 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.742189    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-config-file" (OuterVolumeSpecName: "config-file") pod "105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" (UID: "105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:11:32.743878 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.743855    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-kube-api-access-vhq5h" (OuterVolumeSpecName: "kube-api-access-vhq5h") pod "105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" (UID: "105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328"). InnerVolumeSpecName "kube-api-access-vhq5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:11:32.842983 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.842938    2584 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-config-file\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:11:32.842983 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:32.842976    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhq5h\" (UniqueName: \"kubernetes.io/projected/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328-kube-api-access-vhq5h\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:11:33.301722 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.301690    2584 generic.go:356] "Generic (PLEG): container finished" podID="105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" containerID="3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad" exitCode=0
Jul 14 02:11:33.302231 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.301732    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"
Jul 14 02:11:33.302231 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.301735    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x" event={"ID":"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328","Type":"ContainerDied","Data":"3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad"}
Jul 14 02:11:33.302231 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.301861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bkd6x" event={"ID":"105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328","Type":"ContainerDied","Data":"bf75d574970cdbe5cb89cea254855a01b55568235b989245ed68d974e8cf6814"}
Jul 14 02:11:33.302231 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.301887    2584 scope.go:117] "RemoveContainer" containerID="3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad"
Jul 14 02:11:33.309604 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.309584    2584 scope.go:117] "RemoveContainer" containerID="3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad"
Jul 14 02:11:33.309845 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:11:33.309827    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad\": container with ID starting with 3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad not found: ID does not exist" containerID="3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad"
Jul 14 02:11:33.309908 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.309855    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad"} err="failed to get container status \"3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad\": rpc error: code = NotFound desc = could not find container \"3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad\": container with ID starting with 3324d188ea490f19f39bba9c7e79cf2d9870fde76a874271a542d19e83e5a9ad not found: ID does not exist"
Jul 14 02:11:33.322209 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.319271    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:33.322696 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:33.322679    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bkd6x"]
Jul 14 02:11:34.745283 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.745248    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"]
Jul 14 02:11:34.745726 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.745711    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" containerName="limitador"
Jul 14 02:11:34.745791 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.745730    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" containerName="limitador"
Jul 14 02:11:34.745828 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.745810    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" containerName="limitador"
Jul 14 02:11:34.750579 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.750560    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:34.753155 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.753114    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 14 02:11:34.753299 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.753203    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 02:11:34.753421 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.753364    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dlzvd\""
Jul 14 02:11:34.754348 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.754325    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 02:11:34.755957 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.755938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"]
Jul 14 02:11:34.861195 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.861160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/174b9e78-8a3d-4e45-97d5-bc97feeac0a5-data\") pod \"postgres-69fb4594cf-d7mnn\" (UID: \"174b9e78-8a3d-4e45-97d5-bc97feeac0a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:34.861359 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.861219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vlj\" (UniqueName: \"kubernetes.io/projected/174b9e78-8a3d-4e45-97d5-bc97feeac0a5-kube-api-access-t5vlj\") pod \"postgres-69fb4594cf-d7mnn\" (UID: \"174b9e78-8a3d-4e45-97d5-bc97feeac0a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:34.891189 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.891160    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328" path="/var/lib/kubelet/pods/105eb3cf-f9f8-4fd7-9733-5a6a1bcbc328/volumes"
Jul 14 02:11:34.962472 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.962446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/174b9e78-8a3d-4e45-97d5-bc97feeac0a5-data\") pod \"postgres-69fb4594cf-d7mnn\" (UID: \"174b9e78-8a3d-4e45-97d5-bc97feeac0a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:34.962642 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.962499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vlj\" (UniqueName: \"kubernetes.io/projected/174b9e78-8a3d-4e45-97d5-bc97feeac0a5-kube-api-access-t5vlj\") pod \"postgres-69fb4594cf-d7mnn\" (UID: \"174b9e78-8a3d-4e45-97d5-bc97feeac0a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:34.962876 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.962854    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/174b9e78-8a3d-4e45-97d5-bc97feeac0a5-data\") pod \"postgres-69fb4594cf-d7mnn\" (UID: \"174b9e78-8a3d-4e45-97d5-bc97feeac0a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:34.971739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:34.971714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vlj\" (UniqueName: \"kubernetes.io/projected/174b9e78-8a3d-4e45-97d5-bc97feeac0a5-kube-api-access-t5vlj\") pod \"postgres-69fb4594cf-d7mnn\" (UID: \"174b9e78-8a3d-4e45-97d5-bc97feeac0a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:35.062335 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:35.062241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:35.188067 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:35.188045    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"]
Jul 14 02:11:35.190818 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:11:35.190784    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174b9e78_8a3d_4e45_97d5_bc97feeac0a5.slice/crio-489f02d411ac1ddcf1d9340af0e8654745c38bbdf03b103babefdeba327b5832 WatchSource:0}: Error finding container 489f02d411ac1ddcf1d9340af0e8654745c38bbdf03b103babefdeba327b5832: Status 404 returned error can't find the container with id 489f02d411ac1ddcf1d9340af0e8654745c38bbdf03b103babefdeba327b5832
Jul 14 02:11:35.312391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:35.312310    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn" event={"ID":"174b9e78-8a3d-4e45-97d5-bc97feeac0a5","Type":"ContainerStarted","Data":"489f02d411ac1ddcf1d9340af0e8654745c38bbdf03b103babefdeba327b5832"}
Jul 14 02:11:41.342611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:41.342345    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn" event={"ID":"174b9e78-8a3d-4e45-97d5-bc97feeac0a5","Type":"ContainerStarted","Data":"50e56e4312d685f3a02a5b9235009e1953293b3d083c2684c006dadb0e9a235f"}
Jul 14 02:11:41.342611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:41.342545    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:41.357684 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:41.357633    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn" podStartSLOduration=2.215739944 podStartE2EDuration="7.357618007s" totalImagesPullingTime="5.141878063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:11:41.356521443 +0000 UTC m=+595.029396242" watchObservedRunningTime="2026-07-14 02:11:41.357618007 +0000 UTC m=+595.030492807"
Jul 14 02:11:47.375439 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:47.375411    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d7mnn"
Jul 14 02:11:54.683147 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.683104    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6865cdc9b6-6h8rc"]
Jul 14 02:11:54.705977 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.705947    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6865cdc9b6-6h8rc"]
Jul 14 02:11:54.706118 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.706062    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:54.708912 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.708884    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 02:11:54.708912 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.708891    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 02:11:54.710067 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.710045    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pzgm4\""
Jul 14 02:11:54.817545 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.817512    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6fj\" (UniqueName: \"kubernetes.io/projected/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-kube-api-access-mt6fj\") pod \"maas-controller-6865cdc9b6-6h8rc\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") " pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:54.817712 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.817552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-webhook-cert\") pod \"maas-controller-6865cdc9b6-6h8rc\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") " pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:54.878771 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.878741    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cd5db494-vnrhw"]
Jul 14 02:11:54.880877 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.880861    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:54.892279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.892256    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cd5db494-vnrhw"]
Jul 14 02:11:54.918317 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.918288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31278169-e7ac-4a6f-abd8-0a4045a45cbf-webhook-cert\") pod \"maas-controller-6cd5db494-vnrhw\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") " pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:54.918437 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.918334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6fj\" (UniqueName: \"kubernetes.io/projected/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-kube-api-access-mt6fj\") pod \"maas-controller-6865cdc9b6-6h8rc\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") " pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:54.918437 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.918357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-webhook-cert\") pod \"maas-controller-6865cdc9b6-6h8rc\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") " pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:54.918437 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.918405    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858r6\" (UniqueName: \"kubernetes.io/projected/31278169-e7ac-4a6f-abd8-0a4045a45cbf-kube-api-access-858r6\") pod \"maas-controller-6cd5db494-vnrhw\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") " pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:54.920712 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.920694    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-webhook-cert\") pod \"maas-controller-6865cdc9b6-6h8rc\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") " pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:54.926839 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:54.926812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6fj\" (UniqueName: \"kubernetes.io/projected/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-kube-api-access-mt6fj\") pod \"maas-controller-6865cdc9b6-6h8rc\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") " pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:55.019239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.019152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858r6\" (UniqueName: \"kubernetes.io/projected/31278169-e7ac-4a6f-abd8-0a4045a45cbf-kube-api-access-858r6\") pod \"maas-controller-6cd5db494-vnrhw\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") " pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:55.019239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.019197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31278169-e7ac-4a6f-abd8-0a4045a45cbf-webhook-cert\") pod \"maas-controller-6cd5db494-vnrhw\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") " pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:55.019552 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.019533    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:55.021735 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.021616    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31278169-e7ac-4a6f-abd8-0a4045a45cbf-webhook-cert\") pod \"maas-controller-6cd5db494-vnrhw\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") " pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:55.027626 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.027596    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-858r6\" (UniqueName: \"kubernetes.io/projected/31278169-e7ac-4a6f-abd8-0a4045a45cbf-kube-api-access-858r6\") pod \"maas-controller-6cd5db494-vnrhw\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") " pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:55.144682 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.144658    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6865cdc9b6-6h8rc"]
Jul 14 02:11:55.147094 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:11:55.147058    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd34d7c1_b892_43b8_a2b3_9a59a7edd055.slice/crio-26ce03783f34fda2c8afbb327135de1e3ebb1f717ffbc6c62c5cf3f196fd96e2 WatchSource:0}: Error finding container 26ce03783f34fda2c8afbb327135de1e3ebb1f717ffbc6c62c5cf3f196fd96e2: Status 404 returned error can't find the container with id 26ce03783f34fda2c8afbb327135de1e3ebb1f717ffbc6c62c5cf3f196fd96e2
Jul 14 02:11:55.191663 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.191636    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:55.320199 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.320172    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cd5db494-vnrhw"]
Jul 14 02:11:55.322525 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:11:55.322489    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31278169_e7ac_4a6f_abd8_0a4045a45cbf.slice/crio-7366f475141c898e451688c52cd8d0c8caa535c098e2743dd27a3f372ef0181c WatchSource:0}: Error finding container 7366f475141c898e451688c52cd8d0c8caa535c098e2743dd27a3f372ef0181c: Status 404 returned error can't find the container with id 7366f475141c898e451688c52cd8d0c8caa535c098e2743dd27a3f372ef0181c
Jul 14 02:11:55.400753 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.400717    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc" event={"ID":"fd34d7c1-b892-43b8-a2b3-9a59a7edd055","Type":"ContainerStarted","Data":"26ce03783f34fda2c8afbb327135de1e3ebb1f717ffbc6c62c5cf3f196fd96e2"}
Jul 14 02:11:55.401887 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:55.401861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cd5db494-vnrhw" event={"ID":"31278169-e7ac-4a6f-abd8-0a4045a45cbf","Type":"ContainerStarted","Data":"7366f475141c898e451688c52cd8d0c8caa535c098e2743dd27a3f372ef0181c"}
Jul 14 02:11:57.415879 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:57.415835    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc" event={"ID":"fd34d7c1-b892-43b8-a2b3-9a59a7edd055","Type":"ContainerStarted","Data":"68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6"}
Jul 14 02:11:57.416452 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:57.416425    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:11:57.418071 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:57.418036    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cd5db494-vnrhw" event={"ID":"31278169-e7ac-4a6f-abd8-0a4045a45cbf","Type":"ContainerStarted","Data":"e476c0ee43ca2343cea0d2b48bf13cba0087856a86bbb81e7ef8681f754d9d9f"}
Jul 14 02:11:57.418189 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:57.418175    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:11:57.432629 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:57.432585    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc" podStartSLOduration=1.254968858 podStartE2EDuration="3.432572345s" totalImagesPullingTime="2.177603487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:11:57.431758926 +0000 UTC m=+611.104633727" watchObservedRunningTime="2026-07-14 02:11:57.432572345 +0000 UTC m=+611.105447145"
Jul 14 02:11:57.447995 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:11:57.447949    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cd5db494-vnrhw" podStartSLOduration=1.442003488 podStartE2EDuration="3.447935181s" totalImagesPullingTime="2.005931693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:11:57.446085221 +0000 UTC m=+611.118960024" watchObservedRunningTime="2026-07-14 02:11:57.447935181 +0000 UTC m=+611.120809984"
Jul 14 02:12:08.424176 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.424083    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:12:08.424176 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.424177    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:12:08.507467 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.507433    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6865cdc9b6-6h8rc"]
Jul 14 02:12:08.507713 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.507679    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc" podUID="fd34d7c1-b892-43b8-a2b3-9a59a7edd055" containerName="manager" containerID="cri-o://68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6" gracePeriod=30
Jul 14 02:12:08.756386 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.756361    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:12:08.835180 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.835123    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-webhook-cert\") pod \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") "
Jul 14 02:12:08.835323 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.835215    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt6fj\" (UniqueName: \"kubernetes.io/projected/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-kube-api-access-mt6fj\") pod \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\" (UID: \"fd34d7c1-b892-43b8-a2b3-9a59a7edd055\") "
Jul 14 02:12:08.837210 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.837184    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fd34d7c1-b892-43b8-a2b3-9a59a7edd055" (UID: "fd34d7c1-b892-43b8-a2b3-9a59a7edd055"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:12:08.837377 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.837356    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-kube-api-access-mt6fj" (OuterVolumeSpecName: "kube-api-access-mt6fj") pod "fd34d7c1-b892-43b8-a2b3-9a59a7edd055" (UID: "fd34d7c1-b892-43b8-a2b3-9a59a7edd055"). InnerVolumeSpecName "kube-api-access-mt6fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:12:08.936732 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.936670    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-webhook-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:12:08.936732 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:08.936695    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt6fj\" (UniqueName: \"kubernetes.io/projected/fd34d7c1-b892-43b8-a2b3-9a59a7edd055-kube-api-access-mt6fj\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:12:09.467031 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.466995    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd34d7c1-b892-43b8-a2b3-9a59a7edd055" containerID="68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6" exitCode=0
Jul 14 02:12:09.467031 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.467033    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc"
Jul 14 02:12:09.467490 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.467079    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc" event={"ID":"fd34d7c1-b892-43b8-a2b3-9a59a7edd055","Type":"ContainerDied","Data":"68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6"}
Jul 14 02:12:09.467490 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.467123    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865cdc9b6-6h8rc" event={"ID":"fd34d7c1-b892-43b8-a2b3-9a59a7edd055","Type":"ContainerDied","Data":"26ce03783f34fda2c8afbb327135de1e3ebb1f717ffbc6c62c5cf3f196fd96e2"}
Jul 14 02:12:09.467490 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.467174    2584 scope.go:117] "RemoveContainer" containerID="68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6"
Jul 14 02:12:09.474914 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.474895    2584 scope.go:117] "RemoveContainer" containerID="68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6"
Jul 14 02:12:09.475167 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:12:09.475147    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6\": container with ID starting with 68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6 not found: ID does not exist" containerID="68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6"
Jul 14 02:12:09.475240 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.475173    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6"} err="failed to get container status \"68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6\": rpc error: code = NotFound desc = could not find container \"68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6\": container with ID starting with 68641a8419efcadb502e475ee44cae1c95f1d38abf11b95a35326e9ec78e80f6 not found: ID does not exist"
Jul 14 02:12:09.483762 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.483737    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6865cdc9b6-6h8rc"]
Jul 14 02:12:09.486787 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:09.486768    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6865cdc9b6-6h8rc"]
Jul 14 02:12:10.891296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:10.891258    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd34d7c1-b892-43b8-a2b3-9a59a7edd055" path="/var/lib/kubelet/pods/fd34d7c1-b892-43b8-a2b3-9a59a7edd055/volumes"
Jul 14 02:12:30.765912 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.765876    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"]
Jul 14 02:12:30.766320 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.766175    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd34d7c1-b892-43b8-a2b3-9a59a7edd055" containerName="manager"
Jul 14 02:12:30.766320 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.766187    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd34d7c1-b892-43b8-a2b3-9a59a7edd055" containerName="manager"
Jul 14 02:12:30.766320 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.766250    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd34d7c1-b892-43b8-a2b3-9a59a7edd055" containerName="manager"
Jul 14 02:12:30.772414 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.772394    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:30.777161 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.777142    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cqvjf\""
Jul 14 02:12:30.777642 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.777618    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 14 02:12:30.791408 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.791378    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"]
Jul 14 02:12:30.913833 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.913804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d05146-79a1-4c52-b3e4-d0d6fbc45616-openshift-service-ca\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:30.913989 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.913849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d05146-79a1-4c52-b3e4-d0d6fbc45616-maas-api-tls\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:30.913989 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:30.913914    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4l9\" (UniqueName: \"kubernetes.io/projected/83d05146-79a1-4c52-b3e4-d0d6fbc45616-kube-api-access-ls4l9\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.014682 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.014649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d05146-79a1-4c52-b3e4-d0d6fbc45616-openshift-service-ca\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.014848 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.014711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d05146-79a1-4c52-b3e4-d0d6fbc45616-maas-api-tls\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.014922 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:12:31.014902    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 14 02:12:31.014966 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.014911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4l9\" (UniqueName: \"kubernetes.io/projected/83d05146-79a1-4c52-b3e4-d0d6fbc45616-kube-api-access-ls4l9\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.015007 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:12:31.014978    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83d05146-79a1-4c52-b3e4-d0d6fbc45616-maas-api-tls podName:83d05146-79a1-4c52-b3e4-d0d6fbc45616 nodeName:}" failed. No retries permitted until 2026-07-14 02:12:31.514958267 +0000 UTC m=+645.187833046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83d05146-79a1-4c52-b3e4-d0d6fbc45616-maas-api-tls") pod "maas-api-7d9498bc97-zdqqm" (UID: "83d05146-79a1-4c52-b3e4-d0d6fbc45616") : secret "maas-api-serving-cert" not found
Jul 14 02:12:31.015406 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.015389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d05146-79a1-4c52-b3e4-d0d6fbc45616-openshift-service-ca\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.032737 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.032672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4l9\" (UniqueName: \"kubernetes.io/projected/83d05146-79a1-4c52-b3e4-d0d6fbc45616-kube-api-access-ls4l9\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.483216 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.483179    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"]
Jul 14 02:12:31.487521 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.487502    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.491359 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.491339    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 14 02:12:31.496913 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.496892    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"]
Jul 14 02:12:31.515687 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.515654    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"]
Jul 14 02:12:31.519306 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.519277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d05146-79a1-4c52-b3e4-d0d6fbc45616-maas-api-tls\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.519784 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.519765    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.521801 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.521778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d05146-79a1-4c52-b3e4-d0d6fbc45616-maas-api-tls\") pod \"maas-api-7d9498bc97-zdqqm\" (UID: \"83d05146-79a1-4c52-b3e4-d0d6fbc45616\") " pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.522424 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.522403    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hr8t9\""
Jul 14 02:12:31.530655 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.530634    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"]
Jul 14 02:12:31.620281 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.620247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5e5845-3960-45c9-bd1f-367edcac9090-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-n54pv\" (UID: \"6e5e5845-3960-45c9-bd1f-367edcac9090\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.620281 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.620289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shwd\" (UniqueName: \"kubernetes.io/projected/f1815e0b-97fa-40e7-af2e-39a06174c014-kube-api-access-7shwd\") pod \"payload-processing-f9b8d9b9b-h7wvr\" (UID: \"f1815e0b-97fa-40e7-af2e-39a06174c014\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.620504 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.620311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxj7g\" (UniqueName: \"kubernetes.io/projected/6e5e5845-3960-45c9-bd1f-367edcac9090-kube-api-access-wxj7g\") pod \"payload-pre-processing-74584bb9f8-n54pv\" (UID: \"6e5e5845-3960-45c9-bd1f-367edcac9090\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.620504 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.620334    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1815e0b-97fa-40e7-af2e-39a06174c014-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-h7wvr\" (UID: \"f1815e0b-97fa-40e7-af2e-39a06174c014\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.681852 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.681820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:31.721585 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.720762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shwd\" (UniqueName: \"kubernetes.io/projected/f1815e0b-97fa-40e7-af2e-39a06174c014-kube-api-access-7shwd\") pod \"payload-processing-f9b8d9b9b-h7wvr\" (UID: \"f1815e0b-97fa-40e7-af2e-39a06174c014\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.721585 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.720800    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxj7g\" (UniqueName: \"kubernetes.io/projected/6e5e5845-3960-45c9-bd1f-367edcac9090-kube-api-access-wxj7g\") pod \"payload-pre-processing-74584bb9f8-n54pv\" (UID: \"6e5e5845-3960-45c9-bd1f-367edcac9090\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.721585 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.720838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1815e0b-97fa-40e7-af2e-39a06174c014-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-h7wvr\" (UID: \"f1815e0b-97fa-40e7-af2e-39a06174c014\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.721585 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.720915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5e5845-3960-45c9-bd1f-367edcac9090-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-n54pv\" (UID: \"6e5e5845-3960-45c9-bd1f-367edcac9090\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.721900 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.721760    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5e5845-3960-45c9-bd1f-367edcac9090-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-n54pv\" (UID: \"6e5e5845-3960-45c9-bd1f-367edcac9090\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.722098 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.722074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1815e0b-97fa-40e7-af2e-39a06174c014-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-h7wvr\" (UID: \"f1815e0b-97fa-40e7-af2e-39a06174c014\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.730035 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.730014    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shwd\" (UniqueName: \"kubernetes.io/projected/f1815e0b-97fa-40e7-af2e-39a06174c014-kube-api-access-7shwd\") pod \"payload-processing-f9b8d9b9b-h7wvr\" (UID: \"f1815e0b-97fa-40e7-af2e-39a06174c014\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.730164 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.730066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxj7g\" (UniqueName: \"kubernetes.io/projected/6e5e5845-3960-45c9-bd1f-367edcac9090-kube-api-access-wxj7g\") pod \"payload-pre-processing-74584bb9f8-n54pv\" (UID: \"6e5e5845-3960-45c9-bd1f-367edcac9090\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.798609 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.798574    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:31.830552 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.830503    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"]
Jul 14 02:12:31.831662 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:12:31.831638    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d05146_79a1_4c52_b3e4_d0d6fbc45616.slice/crio-4afaecfce5eff46e2ac0e972715a173963c4919a1b9436bfd350716813b108a5 WatchSource:0}: Error finding container 4afaecfce5eff46e2ac0e972715a173963c4919a1b9436bfd350716813b108a5: Status 404 returned error can't find the container with id 4afaecfce5eff46e2ac0e972715a173963c4919a1b9436bfd350716813b108a5
Jul 14 02:12:31.833214 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.833196    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 02:12:31.838470 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.838446    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:31.956571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:31.956534    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"]
Jul 14 02:12:31.960097 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:12:31.960057    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5e5845_3960_45c9_bd1f_367edcac9090.slice/crio-01c5cdaf7a1f7c6a51cee0b874b5f296cb107928a44724f17c7fda4db146b48b WatchSource:0}: Error finding container 01c5cdaf7a1f7c6a51cee0b874b5f296cb107928a44724f17c7fda4db146b48b: Status 404 returned error can't find the container with id 01c5cdaf7a1f7c6a51cee0b874b5f296cb107928a44724f17c7fda4db146b48b
Jul 14 02:12:32.006266 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:32.006243    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"]
Jul 14 02:12:32.008728 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:12:32.008685    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1815e0b_97fa_40e7_af2e_39a06174c014.slice/crio-590ac82d600b71d02430b0e5b2cefe7600b5a81cea2c5aef21e82b24776608dc WatchSource:0}: Error finding container 590ac82d600b71d02430b0e5b2cefe7600b5a81cea2c5aef21e82b24776608dc: Status 404 returned error can't find the container with id 590ac82d600b71d02430b0e5b2cefe7600b5a81cea2c5aef21e82b24776608dc
Jul 14 02:12:32.569472 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:32.569430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv" event={"ID":"6e5e5845-3960-45c9-bd1f-367edcac9090","Type":"ContainerStarted","Data":"01c5cdaf7a1f7c6a51cee0b874b5f296cb107928a44724f17c7fda4db146b48b"}
Jul 14 02:12:32.570706 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:32.570669    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm" event={"ID":"83d05146-79a1-4c52-b3e4-d0d6fbc45616","Type":"ContainerStarted","Data":"4afaecfce5eff46e2ac0e972715a173963c4919a1b9436bfd350716813b108a5"}
Jul 14 02:12:32.571905 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:32.571881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr" event={"ID":"f1815e0b-97fa-40e7-af2e-39a06174c014","Type":"ContainerStarted","Data":"590ac82d600b71d02430b0e5b2cefe7600b5a81cea2c5aef21e82b24776608dc"}
Jul 14 02:12:34.584470 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.584428    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr" event={"ID":"f1815e0b-97fa-40e7-af2e-39a06174c014","Type":"ContainerStarted","Data":"418df9b383e3f96d773112b1ce4ca677a647001e149499dc66800e9772f92931"}
Jul 14 02:12:34.585049 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.584503    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:34.586120 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.586091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv" event={"ID":"6e5e5845-3960-45c9-bd1f-367edcac9090","Type":"ContainerStarted","Data":"895631dcf561f652931d5df70c12a7c2dea80bb4ff356e6d9f2a41f59b9fc9cd"}
Jul 14 02:12:34.586247 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.586169    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:34.587564 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.587539    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm" event={"ID":"83d05146-79a1-4c52-b3e4-d0d6fbc45616","Type":"ContainerStarted","Data":"76e317f48fc8c18d27ed63fa18e0b113fb50a2a4d51d5f47280aad2eeb60df22"}
Jul 14 02:12:34.587692 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.587674    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:34.600829 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.600788    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr" podStartSLOduration=1.8935680879999999 podStartE2EDuration="3.600770289s" totalImagesPullingTime="1.707202201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:12:34.599896011 +0000 UTC m=+648.272770812" watchObservedRunningTime="2026-07-14 02:12:34.600770289 +0000 UTC m=+648.273645092"
Jul 14 02:12:34.615038 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.614995    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv" podStartSLOduration=1.861458364 podStartE2EDuration="3.614977456s" totalImagesPullingTime="1.753519092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:12:34.614342274 +0000 UTC m=+648.287217065" watchObservedRunningTime="2026-07-14 02:12:34.614977456 +0000 UTC m=+648.287852258"
Jul 14 02:12:34.630478 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:34.630433    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm" podStartSLOduration=2.7478616000000002 podStartE2EDuration="4.630417421s" totalImagesPullingTime="1.882555821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:12:34.630266715 +0000 UTC m=+648.303141517" watchObservedRunningTime="2026-07-14 02:12:34.630417421 +0000 UTC m=+648.303292222"
Jul 14 02:12:40.597468 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:40.597433    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d9498bc97-zdqqm"
Jul 14 02:12:45.594555 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:45.594518    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-h7wvr"
Jul 14 02:12:45.594936 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:45.594808    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-n54pv"
Jul 14 02:12:53.177316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.177286    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"]
Jul 14 02:12:53.181755 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.181739    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.185508 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.185487    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 02:12:53.185508 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.185500    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bxqsh\""
Jul 14 02:12:53.185701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.185500    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 14 02:12:53.185701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.185584    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 02:12:53.190556 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.190533    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"]
Jul 14 02:12:53.199027 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.199001    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xh4t\" (UniqueName: \"kubernetes.io/projected/c1d39986-a273-49b5-9c2b-4edcac6f6d24-kube-api-access-7xh4t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.199179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.199030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d39986-a273-49b5-9c2b-4edcac6f6d24-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.199179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.199050    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.199179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.199115    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.199179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.199158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.199179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.199178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300089 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xh4t\" (UniqueName: \"kubernetes.io/projected/c1d39986-a273-49b5-9c2b-4edcac6f6d24-kube-api-access-7xh4t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d39986-a273-49b5-9c2b-4edcac6f6d24-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300273 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300224    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300562 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300536    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300623 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.300623 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.300611    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.302394 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.302371    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d39986-a273-49b5-9c2b-4edcac6f6d24-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.302628 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.302611    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d39986-a273-49b5-9c2b-4edcac6f6d24-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.307797 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.307771    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xh4t\" (UniqueName: \"kubernetes.io/projected/c1d39986-a273-49b5-9c2b-4edcac6f6d24-kube-api-access-7xh4t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw\" (UID: \"c1d39986-a273-49b5-9c2b-4edcac6f6d24\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.492859 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.492782    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:12:53.621973 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.621939    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"]
Jul 14 02:12:53.625078 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:12:53.625052    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d39986_a273_49b5_9c2b_4edcac6f6d24.slice/crio-4904603336b94324c6d4d91011bf803f2bc0032ffcd572f64661da7f8849bec4 WatchSource:0}: Error finding container 4904603336b94324c6d4d91011bf803f2bc0032ffcd572f64661da7f8849bec4: Status 404 returned error can't find the container with id 4904603336b94324c6d4d91011bf803f2bc0032ffcd572f64661da7f8849bec4
Jul 14 02:12:53.673043 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:53.673007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw" event={"ID":"c1d39986-a273-49b5-9c2b-4edcac6f6d24","Type":"ContainerStarted","Data":"4904603336b94324c6d4d91011bf803f2bc0032ffcd572f64661da7f8849bec4"}
Jul 14 02:12:55.684352 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:55.684315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw" event={"ID":"c1d39986-a273-49b5-9c2b-4edcac6f6d24","Type":"ContainerStarted","Data":"a77189cd1e95b439ef5fdf405d95ff4ed6a400681955d3dcc10f51378e89ade2"}
Jul 14 02:12:55.704359 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:12:55.704311    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw" podStartSLOduration=1.581503982 podStartE2EDuration="2.704297588s" totalImagesPullingTime="1.122793606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:12:55.702714099 +0000 UTC m=+669.375588900" watchObservedRunningTime="2026-07-14 02:12:55.704297588 +0000 UTC m=+669.377172388"
Jul 14 02:13:03.493337 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:03.493297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:13:03.493823 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:03.493366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:13:03.505937 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:03.505914    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:13:03.731548 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:03.731518    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw"
Jul 14 02:13:44.900431 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:44.900359    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"]
Jul 14 02:13:44.902599 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:44.902579    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:44.905088 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:44.905064    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 14 02:13:44.912863 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:44.912840    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"]
Jul 14 02:13:45.027615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.027573    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgd7k\" (UniqueName: \"kubernetes.io/projected/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-kube-api-access-jgd7k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.027615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.027621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.027833 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.027639    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.027833 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.027737    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.027833 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.027780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.027939 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.027852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.128930 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.128890    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.128930 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.128928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129225 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129059    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129225 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129225 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129358 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgd7k\" (UniqueName: \"kubernetes.io/projected/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-kube-api-access-jgd7k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129358 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129427 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.129488 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.129446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.131409 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.131382    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.131575 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.131557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.137148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.137115    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgd7k\" (UniqueName: \"kubernetes.io/projected/d3c80aa9-7909-4cc0-816c-6df79f8bc96b-kube-api-access-jgd7k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg\" (UID: \"d3c80aa9-7909-4cc0-816c-6df79f8bc96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.212405 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.212334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:45.340495 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.340469    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"]
Jul 14 02:13:45.342271 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:13:45.342245    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c80aa9_7909_4cc0_816c_6df79f8bc96b.slice/crio-1eb23fd415103409f682d411fc53d752db9197518cec38fb504cad33e6d59713 WatchSource:0}: Error finding container 1eb23fd415103409f682d411fc53d752db9197518cec38fb504cad33e6d59713: Status 404 returned error can't find the container with id 1eb23fd415103409f682d411fc53d752db9197518cec38fb504cad33e6d59713
Jul 14 02:13:45.893438 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.893403    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg" event={"ID":"d3c80aa9-7909-4cc0-816c-6df79f8bc96b","Type":"ContainerStarted","Data":"798cb0d042cb27ebedcf8f6db9c53b161dce1341ba106b7126a43ea36c65156f"}
Jul 14 02:13:45.893628 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.893449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg" event={"ID":"d3c80aa9-7909-4cc0-816c-6df79f8bc96b","Type":"ContainerStarted","Data":"1eb23fd415103409f682d411fc53d752db9197518cec38fb504cad33e6d59713"}
Jul 14 02:13:45.910206 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:45.910166    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg" podStartSLOduration=1.7437443849999998 podStartE2EDuration="1.910126977s" totalImagesPullingTime="166.382592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:13:45.90909461 +0000 UTC m=+719.581969412" watchObservedRunningTime="2026-07-14 02:13:45.910126977 +0000 UTC m=+719.583001781"
Jul 14 02:13:55.213179 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:55.213151    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:55.213591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:55.213236    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:55.225933 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:55.225911    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:13:55.949221 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:13:55.949193    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg"
Jul 14 02:14:19.205324 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.205293    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"]
Jul 14 02:14:19.207811 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.207783    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.210389 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.210363    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 02:14:19.221855 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.221829    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"]
Jul 14 02:14:19.319196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.319162    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.319351 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.319218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.319351 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.319261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.319351 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.319290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.319459 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.319356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.319459 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.319405    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rsd\" (UniqueName: \"kubernetes.io/projected/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-kube-api-access-b7rsd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420108 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420309 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420309 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420186    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420309 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rsd\" (UniqueName: \"kubernetes.io/projected/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-kube-api-access-b7rsd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420309 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420529 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420529 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420486    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420630 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.420708 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.420685    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.422404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.422384    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.422670 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.422653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.428292 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.428270    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rsd\" (UniqueName: \"kubernetes.io/projected/37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f-kube-api-access-b7rsd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h\" (UID: \"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.517351 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.517280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:19.642634 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:19.642602    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"]
Jul 14 02:14:19.646643 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:14:19.646615    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37bc92f3_f5c6_4ac6_8ef6_ff30aa6d655f.slice/crio-710a29d8d46624ba9e90d41dc91747354a0b6657d6820e78c6bffa775ed005da WatchSource:0}: Error finding container 710a29d8d46624ba9e90d41dc91747354a0b6657d6820e78c6bffa775ed005da: Status 404 returned error can't find the container with id 710a29d8d46624ba9e90d41dc91747354a0b6657d6820e78c6bffa775ed005da
Jul 14 02:14:20.042205 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:20.042163    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h" event={"ID":"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f","Type":"ContainerStarted","Data":"ca1db5858b5c43c298a2094308fe47339e877ece960adddb1549f9e7ea8fc470"}
Jul 14 02:14:20.042205 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:20.042208    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h" event={"ID":"37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f","Type":"ContainerStarted","Data":"710a29d8d46624ba9e90d41dc91747354a0b6657d6820e78c6bffa775ed005da"}
Jul 14 02:14:20.058994 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:20.058944    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h" podStartSLOduration=0.879734127 podStartE2EDuration="1.058930168s" totalImagesPullingTime="179.196041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:14:20.057966658 +0000 UTC m=+753.730841459" watchObservedRunningTime="2026-07-14 02:14:20.058930168 +0000 UTC m=+753.731804968"
Jul 14 02:14:29.517863 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:29.517827    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:29.518279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:29.517898    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:29.530472 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:29.530450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:14:30.096571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:14:30.096545    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h"
Jul 14 02:16:37.425101 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.425067    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cd5db494-vnrhw"]
Jul 14 02:16:37.425657 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.425323    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cd5db494-vnrhw" podUID="31278169-e7ac-4a6f-abd8-0a4045a45cbf" containerName="manager" containerID="cri-o://e476c0ee43ca2343cea0d2b48bf13cba0087856a86bbb81e7ef8681f754d9d9f" gracePeriod=30
Jul 14 02:16:37.625647 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.625618    2584 generic.go:356] "Generic (PLEG): container finished" podID="31278169-e7ac-4a6f-abd8-0a4045a45cbf" containerID="e476c0ee43ca2343cea0d2b48bf13cba0087856a86bbb81e7ef8681f754d9d9f" exitCode=0
Jul 14 02:16:37.625784 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.625689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cd5db494-vnrhw" event={"ID":"31278169-e7ac-4a6f-abd8-0a4045a45cbf","Type":"ContainerDied","Data":"e476c0ee43ca2343cea0d2b48bf13cba0087856a86bbb81e7ef8681f754d9d9f"}
Jul 14 02:16:37.682214 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.682153    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:16:37.779798 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.779770    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-858r6\" (UniqueName: \"kubernetes.io/projected/31278169-e7ac-4a6f-abd8-0a4045a45cbf-kube-api-access-858r6\") pod \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") "
Jul 14 02:16:37.779966 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.779885    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31278169-e7ac-4a6f-abd8-0a4045a45cbf-webhook-cert\") pod \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\" (UID: \"31278169-e7ac-4a6f-abd8-0a4045a45cbf\") "
Jul 14 02:16:37.781611 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.781577    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31278169-e7ac-4a6f-abd8-0a4045a45cbf-kube-api-access-858r6" (OuterVolumeSpecName: "kube-api-access-858r6") pod "31278169-e7ac-4a6f-abd8-0a4045a45cbf" (UID: "31278169-e7ac-4a6f-abd8-0a4045a45cbf"). InnerVolumeSpecName "kube-api-access-858r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:16:37.781780 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.781760    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31278169-e7ac-4a6f-abd8-0a4045a45cbf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "31278169-e7ac-4a6f-abd8-0a4045a45cbf" (UID: "31278169-e7ac-4a6f-abd8-0a4045a45cbf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:16:37.880494 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.880463    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858r6\" (UniqueName: \"kubernetes.io/projected/31278169-e7ac-4a6f-abd8-0a4045a45cbf-kube-api-access-858r6\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:16:37.880494 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:37.880490    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31278169-e7ac-4a6f-abd8-0a4045a45cbf-webhook-cert\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:16:38.631526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:38.631500    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cd5db494-vnrhw"
Jul 14 02:16:38.631966 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:38.631500    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cd5db494-vnrhw" event={"ID":"31278169-e7ac-4a6f-abd8-0a4045a45cbf","Type":"ContainerDied","Data":"7366f475141c898e451688c52cd8d0c8caa535c098e2743dd27a3f372ef0181c"}
Jul 14 02:16:38.631966 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:38.631630    2584 scope.go:117] "RemoveContainer" containerID="e476c0ee43ca2343cea0d2b48bf13cba0087856a86bbb81e7ef8681f754d9d9f"
Jul 14 02:16:38.651397 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:38.651374    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cd5db494-vnrhw"]
Jul 14 02:16:38.655630 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:38.655608    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cd5db494-vnrhw"]
Jul 14 02:16:38.890944 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:38.890918    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31278169-e7ac-4a6f-abd8-0a4045a45cbf" path="/var/lib/kubelet/pods/31278169-e7ac-4a6f-abd8-0a4045a45cbf/volumes"
Jul 14 02:16:39.422074 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.422040    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cd5db494-ctxrl"]
Jul 14 02:16:39.422405 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.422392    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31278169-e7ac-4a6f-abd8-0a4045a45cbf" containerName="manager"
Jul 14 02:16:39.422458 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.422406    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="31278169-e7ac-4a6f-abd8-0a4045a45cbf" containerName="manager"
Jul 14 02:16:39.422492 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.422471    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="31278169-e7ac-4a6f-abd8-0a4045a45cbf" containerName="manager"
Jul 14 02:16:39.425027 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.425009    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.427644 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.427626    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 02:16:39.428816 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.428794    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pzgm4\""
Jul 14 02:16:39.428896 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.428828    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 02:16:39.438638 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.438610    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cd5db494-ctxrl"]
Jul 14 02:16:39.494605 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.494577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c111897d-f0ea-4300-bc6a-be33476a0285-webhook-cert\") pod \"maas-controller-6cd5db494-ctxrl\" (UID: \"c111897d-f0ea-4300-bc6a-be33476a0285\") " pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.494743 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.494622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwnx\" (UniqueName: \"kubernetes.io/projected/c111897d-f0ea-4300-bc6a-be33476a0285-kube-api-access-wvwnx\") pod \"maas-controller-6cd5db494-ctxrl\" (UID: \"c111897d-f0ea-4300-bc6a-be33476a0285\") " pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.595229 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.595199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwnx\" (UniqueName: \"kubernetes.io/projected/c111897d-f0ea-4300-bc6a-be33476a0285-kube-api-access-wvwnx\") pod \"maas-controller-6cd5db494-ctxrl\" (UID: \"c111897d-f0ea-4300-bc6a-be33476a0285\") " pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.595385 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.595306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c111897d-f0ea-4300-bc6a-be33476a0285-webhook-cert\") pod \"maas-controller-6cd5db494-ctxrl\" (UID: \"c111897d-f0ea-4300-bc6a-be33476a0285\") " pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.597609 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.597581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c111897d-f0ea-4300-bc6a-be33476a0285-webhook-cert\") pod \"maas-controller-6cd5db494-ctxrl\" (UID: \"c111897d-f0ea-4300-bc6a-be33476a0285\") " pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.602594 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.602572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwnx\" (UniqueName: \"kubernetes.io/projected/c111897d-f0ea-4300-bc6a-be33476a0285-kube-api-access-wvwnx\") pod \"maas-controller-6cd5db494-ctxrl\" (UID: \"c111897d-f0ea-4300-bc6a-be33476a0285\") " pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.736280 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.736188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:39.867346 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:39.867318    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cd5db494-ctxrl"]
Jul 14 02:16:39.869487 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:16:39.869464    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc111897d_f0ea_4300_bc6a_be33476a0285.slice/crio-cc22028393b629feb98431239808cd3e37ef3bc733fa7708620a72c9d9db80c1 WatchSource:0}: Error finding container cc22028393b629feb98431239808cd3e37ef3bc733fa7708620a72c9d9db80c1: Status 404 returned error can't find the container with id cc22028393b629feb98431239808cd3e37ef3bc733fa7708620a72c9d9db80c1
Jul 14 02:16:40.642146 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:40.642103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cd5db494-ctxrl" event={"ID":"c111897d-f0ea-4300-bc6a-be33476a0285","Type":"ContainerStarted","Data":"e1c1cf142908a5fa7967f5f32813b1906bfbcee491dec5a3303c5cca5787c97f"}
Jul 14 02:16:40.642303 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:40.642175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cd5db494-ctxrl" event={"ID":"c111897d-f0ea-4300-bc6a-be33476a0285","Type":"ContainerStarted","Data":"cc22028393b629feb98431239808cd3e37ef3bc733fa7708620a72c9d9db80c1"}
Jul 14 02:16:40.642303 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:40.642222    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:16:40.659886 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:40.659840    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cd5db494-ctxrl" podStartSLOduration=1.057082181 podStartE2EDuration="1.659826421s" totalImagesPullingTime="602.74424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:16:40.658201235 +0000 UTC m=+894.331076034" watchObservedRunningTime="2026-07-14 02:16:40.659826421 +0000 UTC m=+894.332701220"
Jul 14 02:16:51.649699 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:16:51.649669    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cd5db494-ctxrl"
Jul 14 02:28:47.403057 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.403027    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"]
Jul 14 02:28:47.406291 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.406272    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.409402 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.409379    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4535243d-serving-cert\""
Jul 14 02:28:47.431171 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.431121    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"]
Jul 14 02:28:47.568752 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.568720    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-openshift-service-ca\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.568929 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.568769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-maas-api-tls\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.568929 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.568812    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zb7\" (UniqueName: \"kubernetes.io/projected/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-kube-api-access-b7zb7\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.669337 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.669249    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-openshift-service-ca\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.669337 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.669309    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-maas-api-tls\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.669531 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.669374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zb7\" (UniqueName: \"kubernetes.io/projected/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-kube-api-access-b7zb7\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.669942 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.669916    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-openshift-service-ca\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.671818 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.671795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-maas-api-tls\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.676949 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.676926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zb7\" (UniqueName: \"kubernetes.io/projected/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-kube-api-access-b7zb7\") pod \"maas-api-e2e-ait-4535243d-b57485f76-bt5tn\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.715753 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.715722    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:47.847619 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.847592    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"]
Jul 14 02:28:47.849258 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:28:47.849232    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e9aed9_8b12_4ca8_b45b_94ad77a397ed.slice/crio-5adadf4450404daaf436668e10e8bbd179e5a8bca80cd6676b355f9413324a98 WatchSource:0}: Error finding container 5adadf4450404daaf436668e10e8bbd179e5a8bca80cd6676b355f9413324a98: Status 404 returned error can't find the container with id 5adadf4450404daaf436668e10e8bbd179e5a8bca80cd6676b355f9413324a98
Jul 14 02:28:47.850564 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:47.850541    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 02:28:48.844158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:48.844109    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn" event={"ID":"36e9aed9-8b12-4ca8-b45b-94ad77a397ed","Type":"ContainerStarted","Data":"63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e"}
Jul 14 02:28:48.844586 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:48.844221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn" event={"ID":"36e9aed9-8b12-4ca8-b45b-94ad77a397ed","Type":"ContainerStarted","Data":"5adadf4450404daaf436668e10e8bbd179e5a8bca80cd6676b355f9413324a98"}
Jul 14 02:28:48.844586 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:48.844248    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:28:48.861959 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:48.861911    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn" podStartSLOduration=1.406947358 podStartE2EDuration="1.861897044s" totalImagesPullingTime="454.949686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:28:48.859507975 +0000 UTC m=+1622.532382776" watchObservedRunningTime="2026-07-14 02:28:48.861897044 +0000 UTC m=+1622.534771844"
Jul 14 02:28:54.853217 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:28:54.853188    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:29:52.866326 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:52.866290    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"]
Jul 14 02:29:52.866942 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:52.866567    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn" podUID="36e9aed9-8b12-4ca8-b45b-94ad77a397ed" containerName="maas-api" containerID="cri-o://63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e" gracePeriod=30
Jul 14 02:29:53.115006 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.114981    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:29:53.118571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.118512    2584 generic.go:356] "Generic (PLEG): container finished" podID="36e9aed9-8b12-4ca8-b45b-94ad77a397ed" containerID="63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e" exitCode=0
Jul 14 02:29:53.118571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.118547    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"
Jul 14 02:29:53.118571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.118562    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn" event={"ID":"36e9aed9-8b12-4ca8-b45b-94ad77a397ed","Type":"ContainerDied","Data":"63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e"}
Jul 14 02:29:53.118739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.118604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn" event={"ID":"36e9aed9-8b12-4ca8-b45b-94ad77a397ed","Type":"ContainerDied","Data":"5adadf4450404daaf436668e10e8bbd179e5a8bca80cd6676b355f9413324a98"}
Jul 14 02:29:53.118739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.118627    2584 scope.go:117] "RemoveContainer" containerID="63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e"
Jul 14 02:29:53.126389 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.126368    2584 scope.go:117] "RemoveContainer" containerID="63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e"
Jul 14 02:29:53.126682 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:29:53.126660    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e\": container with ID starting with 63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e not found: ID does not exist" containerID="63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e"
Jul 14 02:29:53.126768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.126689    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e"} err="failed to get container status \"63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e\": rpc error: code = NotFound desc = could not find container \"63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e\": container with ID starting with 63ff91dca370c8683554eed185a1deea217f5822dd91d41d626bcfb89f48e74e not found: ID does not exist"
Jul 14 02:29:53.223148 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.223097    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-openshift-service-ca\") pod \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") "
Jul 14 02:29:53.223307 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.223164    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-maas-api-tls\") pod \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") "
Jul 14 02:29:53.223307 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.223202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7zb7\" (UniqueName: \"kubernetes.io/projected/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-kube-api-access-b7zb7\") pod \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\" (UID: \"36e9aed9-8b12-4ca8-b45b-94ad77a397ed\") "
Jul 14 02:29:53.223541 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.223513    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36e9aed9-8b12-4ca8-b45b-94ad77a397ed" (UID: "36e9aed9-8b12-4ca8-b45b-94ad77a397ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:29:53.225366 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.225343    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-kube-api-access-b7zb7" (OuterVolumeSpecName: "kube-api-access-b7zb7") pod "36e9aed9-8b12-4ca8-b45b-94ad77a397ed" (UID: "36e9aed9-8b12-4ca8-b45b-94ad77a397ed"). InnerVolumeSpecName "kube-api-access-b7zb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:29:53.225443 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.225423    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36e9aed9-8b12-4ca8-b45b-94ad77a397ed" (UID: "36e9aed9-8b12-4ca8-b45b-94ad77a397ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:29:53.324575 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.324540    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:29:53.324575 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.324567    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zb7\" (UniqueName: \"kubernetes.io/projected/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-kube-api-access-b7zb7\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:29:53.324575 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.324579    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e9aed9-8b12-4ca8-b45b-94ad77a397ed-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:29:53.439379 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.439351    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"]
Jul 14 02:29:53.442572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:53.442547    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4535243d-b57485f76-bt5tn"]
Jul 14 02:29:54.891630 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:29:54.891589    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36e9aed9-8b12-4ca8-b45b-94ad77a397ed" path="/var/lib/kubelet/pods/36e9aed9-8b12-4ca8-b45b-94ad77a397ed/volumes"
Jul 14 02:30:09.632474 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.632438    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"]
Jul 14 02:30:09.632997 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.632978    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36e9aed9-8b12-4ca8-b45b-94ad77a397ed" containerName="maas-api"
Jul 14 02:30:09.633048 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.633001    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36e9aed9-8b12-4ca8-b45b-94ad77a397ed" containerName="maas-api"
Jul 14 02:30:09.633111 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.633099    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36e9aed9-8b12-4ca8-b45b-94ad77a397ed" containerName="maas-api"
Jul 14 02:30:09.637753 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.637732    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.640378 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.640352    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fc424a89-serving-cert\""
Jul 14 02:30:09.645196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.645174    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"]
Jul 14 02:30:09.759071 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.759035    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ft9\" (UniqueName: \"kubernetes.io/projected/97c391df-e4f7-435a-80a9-66e5111237fd-kube-api-access-m5ft9\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.759264 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.759094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c391df-e4f7-435a-80a9-66e5111237fd-maas-api-tls\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.759264 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.759191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c391df-e4f7-435a-80a9-66e5111237fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.859748 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.859715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c391df-e4f7-435a-80a9-66e5111237fd-maas-api-tls\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.859908 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.859755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c391df-e4f7-435a-80a9-66e5111237fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.859908 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.859819    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ft9\" (UniqueName: \"kubernetes.io/projected/97c391df-e4f7-435a-80a9-66e5111237fd-kube-api-access-m5ft9\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.860568 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.860544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c391df-e4f7-435a-80a9-66e5111237fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.862180 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.862157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c391df-e4f7-435a-80a9-66e5111237fd-maas-api-tls\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.871271 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.871251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ft9\" (UniqueName: \"kubernetes.io/projected/97c391df-e4f7-435a-80a9-66e5111237fd-kube-api-access-m5ft9\") pod \"maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:09.950272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:09.950204    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:10.081045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:10.080929    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"]
Jul 14 02:30:10.083348 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:30:10.083321    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c391df_e4f7_435a_80a9_66e5111237fd.slice/crio-6f7f4d424918af5c86ddd21c3958400e3f9c84fc5ae535f856e4756a59666d7b WatchSource:0}: Error finding container 6f7f4d424918af5c86ddd21c3958400e3f9c84fc5ae535f856e4756a59666d7b: Status 404 returned error can't find the container with id 6f7f4d424918af5c86ddd21c3958400e3f9c84fc5ae535f856e4756a59666d7b
Jul 14 02:30:10.203377 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:10.203288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj" event={"ID":"97c391df-e4f7-435a-80a9-66e5111237fd","Type":"ContainerStarted","Data":"6f7f4d424918af5c86ddd21c3958400e3f9c84fc5ae535f856e4756a59666d7b"}
Jul 14 02:30:11.214856 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:11.214819    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj" event={"ID":"97c391df-e4f7-435a-80a9-66e5111237fd","Type":"ContainerStarted","Data":"a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382"}
Jul 14 02:30:11.215312 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:11.214929    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:30:11.231321 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:11.231277    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj" podStartSLOduration=1.762557759 podStartE2EDuration="2.231263452s" totalImagesPullingTime="468.705693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:30:11.229971504 +0000 UTC m=+1704.902846315" watchObservedRunningTime="2026-07-14 02:30:11.231263452 +0000 UTC m=+1704.904138252"
Jul 14 02:30:17.224553 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:30:17.224527    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:31:19.760789 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:19.760708    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"]
Jul 14 02:31:19.761299 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:19.761085    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj" podUID="97c391df-e4f7-435a-80a9-66e5111237fd" containerName="maas-api" containerID="cri-o://a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382" gracePeriod=30
Jul 14 02:31:20.006697 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.006675    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:31:20.153786 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.153756    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c391df-e4f7-435a-80a9-66e5111237fd-maas-api-tls\") pod \"97c391df-e4f7-435a-80a9-66e5111237fd\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") "
Jul 14 02:31:20.153969 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.153791    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c391df-e4f7-435a-80a9-66e5111237fd-openshift-service-ca\") pod \"97c391df-e4f7-435a-80a9-66e5111237fd\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") "
Jul 14 02:31:20.153969 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.153844    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5ft9\" (UniqueName: \"kubernetes.io/projected/97c391df-e4f7-435a-80a9-66e5111237fd-kube-api-access-m5ft9\") pod \"97c391df-e4f7-435a-80a9-66e5111237fd\" (UID: \"97c391df-e4f7-435a-80a9-66e5111237fd\") "
Jul 14 02:31:20.154202 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.154174    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c391df-e4f7-435a-80a9-66e5111237fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97c391df-e4f7-435a-80a9-66e5111237fd" (UID: "97c391df-e4f7-435a-80a9-66e5111237fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:31:20.155834 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.155812    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c391df-e4f7-435a-80a9-66e5111237fd-kube-api-access-m5ft9" (OuterVolumeSpecName: "kube-api-access-m5ft9") pod "97c391df-e4f7-435a-80a9-66e5111237fd" (UID: "97c391df-e4f7-435a-80a9-66e5111237fd"). InnerVolumeSpecName "kube-api-access-m5ft9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:31:20.155903 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.155830    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c391df-e4f7-435a-80a9-66e5111237fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97c391df-e4f7-435a-80a9-66e5111237fd" (UID: "97c391df-e4f7-435a-80a9-66e5111237fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:31:20.255279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.255247    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ft9\" (UniqueName: \"kubernetes.io/projected/97c391df-e4f7-435a-80a9-66e5111237fd-kube-api-access-m5ft9\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:31:20.255279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.255277    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c391df-e4f7-435a-80a9-66e5111237fd-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:31:20.255279 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.255286    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c391df-e4f7-435a-80a9-66e5111237fd-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:31:20.519729 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.519648    2584 generic.go:356] "Generic (PLEG): container finished" podID="97c391df-e4f7-435a-80a9-66e5111237fd" containerID="a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382" exitCode=0
Jul 14 02:31:20.519729 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.519691    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"
Jul 14 02:31:20.519729 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.519694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj" event={"ID":"97c391df-e4f7-435a-80a9-66e5111237fd","Type":"ContainerDied","Data":"a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382"}
Jul 14 02:31:20.519941 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.519758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj" event={"ID":"97c391df-e4f7-435a-80a9-66e5111237fd","Type":"ContainerDied","Data":"6f7f4d424918af5c86ddd21c3958400e3f9c84fc5ae535f856e4756a59666d7b"}
Jul 14 02:31:20.519941 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.519784    2584 scope.go:117] "RemoveContainer" containerID="a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382"
Jul 14 02:31:20.528678 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.528660    2584 scope.go:117] "RemoveContainer" containerID="a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382"
Jul 14 02:31:20.528930 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:31:20.528915    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382\": container with ID starting with a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382 not found: ID does not exist" containerID="a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382"
Jul 14 02:31:20.528995 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.528941    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382"} err="failed to get container status \"a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382\": rpc error: code = NotFound desc = could not find container \"a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382\": container with ID starting with a275e8731158fae76a26eee9593bb3c9b642311b81e0ba396e67c881ce6d4382 not found: ID does not exist"
Jul 14 02:31:20.539620 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.539596    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"]
Jul 14 02:31:20.543298 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.543278    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc424a89-646459fd68-wvxrj"]
Jul 14 02:31:20.891619 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:20.891588    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c391df-e4f7-435a-80a9-66e5111237fd" path="/var/lib/kubelet/pods/97c391df-e4f7-435a-80a9-66e5111237fd/volumes"
Jul 14 02:31:26.954290 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.954250    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"]
Jul 14 02:31:26.954828 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.954795    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c391df-e4f7-435a-80a9-66e5111237fd" containerName="maas-api"
Jul 14 02:31:26.954828 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.954819    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c391df-e4f7-435a-80a9-66e5111237fd" containerName="maas-api"
Jul 14 02:31:26.955021 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.954926    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c391df-e4f7-435a-80a9-66e5111237fd" containerName="maas-api"
Jul 14 02:31:26.959039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.959018    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:26.961631 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.961606    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b25e0f88-serving-cert\""
Jul 14 02:31:26.964559 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:26.964536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"]
Jul 14 02:31:27.005657 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.005624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvgb\" (UniqueName: \"kubernetes.io/projected/65e76995-8615-4f4c-be75-a7c67ed02cfb-kube-api-access-qwvgb\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.005804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.005685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e76995-8615-4f4c-be75-a7c67ed02cfb-openshift-service-ca\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.005804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.005707    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.106058 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.106021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvgb\" (UniqueName: \"kubernetes.io/projected/65e76995-8615-4f4c-be75-a7c67ed02cfb-kube-api-access-qwvgb\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.106239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.106084    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e76995-8615-4f4c-be75-a7c67ed02cfb-openshift-service-ca\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.106239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.106113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.106313 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:31:27.106249    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-serving-cert: secret "maas-api-e2e-ait-b25e0f88-serving-cert" not found
Jul 14 02:31:27.106401 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:31:27.106318    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls podName:65e76995-8615-4f4c-be75-a7c67ed02cfb nodeName:}" failed. No retries permitted until 2026-07-14 02:31:27.606300662 +0000 UTC m=+1781.279175439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls") pod "maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" (UID: "65e76995-8615-4f4c-be75-a7c67ed02cfb") : secret "maas-api-e2e-ait-b25e0f88-serving-cert" not found
Jul 14 02:31:27.106745 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.106726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e76995-8615-4f4c-be75-a7c67ed02cfb-openshift-service-ca\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.116703 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.116680    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvgb\" (UniqueName: \"kubernetes.io/projected/65e76995-8615-4f4c-be75-a7c67ed02cfb-kube-api-access-qwvgb\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.610754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.610719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.613075 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.613052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls\") pod \"maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:27.870765 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:27.870667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:28.006527 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:28.006498    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"]
Jul 14 02:31:28.008724 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:31:28.008694    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e76995_8615_4f4c_be75_a7c67ed02cfb.slice/crio-b7723d9dc631a3ae97fc7caf779a58357165fc85af81acc16de62128132e537d WatchSource:0}: Error finding container b7723d9dc631a3ae97fc7caf779a58357165fc85af81acc16de62128132e537d: Status 404 returned error can't find the container with id b7723d9dc631a3ae97fc7caf779a58357165fc85af81acc16de62128132e537d
Jul 14 02:31:28.556289 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:28.556252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" event={"ID":"65e76995-8615-4f4c-be75-a7c67ed02cfb","Type":"ContainerStarted","Data":"f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348"}
Jul 14 02:31:28.556473 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:28.556301    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" event={"ID":"65e76995-8615-4f4c-be75-a7c67ed02cfb","Type":"ContainerStarted","Data":"b7723d9dc631a3ae97fc7caf779a58357165fc85af81acc16de62128132e537d"}
Jul 14 02:31:28.556473 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:28.556331    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:31:28.573879 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:28.573836    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" podStartSLOduration=2.203601298 podStartE2EDuration="2.573822442s" totalImagesPullingTime="370.221144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:31:28.572055561 +0000 UTC m=+1782.244930361" watchObservedRunningTime="2026-07-14 02:31:28.573822442 +0000 UTC m=+1782.246697286"
Jul 14 02:31:34.566629 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:31:34.566596    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:32:42.337249 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.337213    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"]
Jul 14 02:32:42.337843 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.337467    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" podUID="65e76995-8615-4f4c-be75-a7c67ed02cfb" containerName="maas-api" containerID="cri-o://f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348" gracePeriod=30
Jul 14 02:32:42.577858 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.577834    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:32:42.651971 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.651947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls\") pod \"65e76995-8615-4f4c-be75-a7c67ed02cfb\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") "
Jul 14 02:32:42.652116 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.652026    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwvgb\" (UniqueName: \"kubernetes.io/projected/65e76995-8615-4f4c-be75-a7c67ed02cfb-kube-api-access-qwvgb\") pod \"65e76995-8615-4f4c-be75-a7c67ed02cfb\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") "
Jul 14 02:32:42.652116 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.652099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e76995-8615-4f4c-be75-a7c67ed02cfb-openshift-service-ca\") pod \"65e76995-8615-4f4c-be75-a7c67ed02cfb\" (UID: \"65e76995-8615-4f4c-be75-a7c67ed02cfb\") "
Jul 14 02:32:42.652451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.652426    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e76995-8615-4f4c-be75-a7c67ed02cfb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65e76995-8615-4f4c-be75-a7c67ed02cfb" (UID: "65e76995-8615-4f4c-be75-a7c67ed02cfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:32:42.653943 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.653918    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65e76995-8615-4f4c-be75-a7c67ed02cfb" (UID: "65e76995-8615-4f4c-be75-a7c67ed02cfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:32:42.654039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.654021    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e76995-8615-4f4c-be75-a7c67ed02cfb-kube-api-access-qwvgb" (OuterVolumeSpecName: "kube-api-access-qwvgb") pod "65e76995-8615-4f4c-be75-a7c67ed02cfb" (UID: "65e76995-8615-4f4c-be75-a7c67ed02cfb"). InnerVolumeSpecName "kube-api-access-qwvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:32:42.753158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.753102    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvgb\" (UniqueName: \"kubernetes.io/projected/65e76995-8615-4f4c-be75-a7c67ed02cfb-kube-api-access-qwvgb\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:32:42.753158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.753149    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e76995-8615-4f4c-be75-a7c67ed02cfb-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:32:42.753158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.753164    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65e76995-8615-4f4c-be75-a7c67ed02cfb-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:32:42.885076 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.885040    2584 generic.go:356] "Generic (PLEG): container finished" podID="65e76995-8615-4f4c-be75-a7c67ed02cfb" containerID="f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348" exitCode=0
Jul 14 02:32:42.885294 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.885088    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"
Jul 14 02:32:42.885294 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.885094    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" event={"ID":"65e76995-8615-4f4c-be75-a7c67ed02cfb","Type":"ContainerDied","Data":"f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348"}
Jul 14 02:32:42.885294 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.885185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt" event={"ID":"65e76995-8615-4f4c-be75-a7c67ed02cfb","Type":"ContainerDied","Data":"b7723d9dc631a3ae97fc7caf779a58357165fc85af81acc16de62128132e537d"}
Jul 14 02:32:42.885294 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.885207    2584 scope.go:117] "RemoveContainer" containerID="f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348"
Jul 14 02:32:42.894421 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.894401    2584 scope.go:117] "RemoveContainer" containerID="f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348"
Jul 14 02:32:42.894668 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:32:42.894647    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348\": container with ID starting with f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348 not found: ID does not exist" containerID="f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348"
Jul 14 02:32:42.894730 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.894676    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348"} err="failed to get container status \"f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348\": rpc error: code = NotFound desc = could not find container \"f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348\": container with ID starting with f94d7b4bd09e82c30c08bbe8d80a6c0c0e9fb59fd2dfee06ed6683319c2af348 not found: ID does not exist"
Jul 14 02:32:42.908163 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.908069    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"]
Jul 14 02:32:42.909694 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:42.909674    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b25e0f88-59f9dc76b-dbmpt"]
Jul 14 02:32:44.891985 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:44.891953    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e76995-8615-4f4c-be75-a7c67ed02cfb" path="/var/lib/kubelet/pods/65e76995-8615-4f4c-be75-a7c67ed02cfb/volumes"
Jul 14 02:32:59.578170 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.578116    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"]
Jul 14 02:32:59.578642 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.578489    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e76995-8615-4f4c-be75-a7c67ed02cfb" containerName="maas-api"
Jul 14 02:32:59.578642 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.578505    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e76995-8615-4f4c-be75-a7c67ed02cfb" containerName="maas-api"
Jul 14 02:32:59.578642 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.578587    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e76995-8615-4f4c-be75-a7c67ed02cfb" containerName="maas-api"
Jul 14 02:32:59.588323 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.588296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.590797 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.590767    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"]
Jul 14 02:32:59.590946 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.590927    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a5aba0cb-serving-cert\""
Jul 14 02:32:59.703150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.703093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97z9\" (UniqueName: \"kubernetes.io/projected/bda8a039-b607-4490-947b-4bcc2ce9ee32-kube-api-access-x97z9\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.703318 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.703163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda8a039-b607-4490-947b-4bcc2ce9ee32-openshift-service-ca\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.703318 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.703292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.804043 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.804004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.804043 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.804048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97z9\" (UniqueName: \"kubernetes.io/projected/bda8a039-b607-4490-947b-4bcc2ce9ee32-kube-api-access-x97z9\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.804296 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.804075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda8a039-b607-4490-947b-4bcc2ce9ee32-openshift-service-ca\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.804296 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:32:59.804190    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-serving-cert: secret "maas-api-e2e-derive-a5aba0cb-serving-cert" not found
Jul 14 02:32:59.804296 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:32:59.804266    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls podName:bda8a039-b607-4490-947b-4bcc2ce9ee32 nodeName:}" failed. No retries permitted until 2026-07-14 02:33:00.304249717 +0000 UTC m=+1873.977124498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls") pod "maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" (UID: "bda8a039-b607-4490-947b-4bcc2ce9ee32") : secret "maas-api-e2e-derive-a5aba0cb-serving-cert" not found
Jul 14 02:32:59.804740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.804723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda8a039-b607-4490-947b-4bcc2ce9ee32-openshift-service-ca\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:32:59.812579 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:32:59.812554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97z9\" (UniqueName: \"kubernetes.io/projected/bda8a039-b607-4490-947b-4bcc2ce9ee32-kube-api-access-x97z9\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:33:00.308373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:00.308339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:33:00.310742 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:00.310714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls\") pod \"maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:33:00.501193 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:00.501159    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:33:00.633354 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:00.633330    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"]
Jul 14 02:33:00.635955 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:33:00.635917    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda8a039_b607_4490_947b_4bcc2ce9ee32.slice/crio-e73ebec67809c73218ddd6a67980c8b7f6cd27dd427c6fcbb214d23ec8a368ce WatchSource:0}: Error finding container e73ebec67809c73218ddd6a67980c8b7f6cd27dd427c6fcbb214d23ec8a368ce: Status 404 returned error can't find the container with id e73ebec67809c73218ddd6a67980c8b7f6cd27dd427c6fcbb214d23ec8a368ce
Jul 14 02:33:00.965385 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:00.965347    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" event={"ID":"bda8a039-b607-4490-947b-4bcc2ce9ee32","Type":"ContainerStarted","Data":"e73ebec67809c73218ddd6a67980c8b7f6cd27dd427c6fcbb214d23ec8a368ce"}
Jul 14 02:33:01.972031 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:01.971999    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" event={"ID":"bda8a039-b607-4490-947b-4bcc2ce9ee32","Type":"ContainerStarted","Data":"a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47"}
Jul 14 02:33:01.972448 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:01.972140    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:33:01.988504 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:01.988458    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" podStartSLOduration=2.576961267 podStartE2EDuration="2.988443662s" totalImagesPullingTime="411.482395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:33:01.986783356 +0000 UTC m=+1875.659658173" watchObservedRunningTime="2026-07-14 02:33:01.988443662 +0000 UTC m=+1875.661318462"
Jul 14 02:33:07.982996 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:33:07.982966    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:34:04.278507 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.278335    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"]
Jul 14 02:34:04.279021 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.278577    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" podUID="bda8a039-b607-4490-947b-4bcc2ce9ee32" containerName="maas-api" containerID="cri-o://a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47" gracePeriod=30
Jul 14 02:34:04.527996 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.527966    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:34:04.636575 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.636546    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls\") pod \"bda8a039-b607-4490-947b-4bcc2ce9ee32\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") "
Jul 14 02:34:04.636754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.636613    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda8a039-b607-4490-947b-4bcc2ce9ee32-openshift-service-ca\") pod \"bda8a039-b607-4490-947b-4bcc2ce9ee32\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") "
Jul 14 02:34:04.636754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.636634    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x97z9\" (UniqueName: \"kubernetes.io/projected/bda8a039-b607-4490-947b-4bcc2ce9ee32-kube-api-access-x97z9\") pod \"bda8a039-b607-4490-947b-4bcc2ce9ee32\" (UID: \"bda8a039-b607-4490-947b-4bcc2ce9ee32\") "
Jul 14 02:34:04.636993 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.636967    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda8a039-b607-4490-947b-4bcc2ce9ee32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bda8a039-b607-4490-947b-4bcc2ce9ee32" (UID: "bda8a039-b607-4490-947b-4bcc2ce9ee32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:34:04.638736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.638707    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bda8a039-b607-4490-947b-4bcc2ce9ee32" (UID: "bda8a039-b607-4490-947b-4bcc2ce9ee32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:34:04.638835 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.638757    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda8a039-b607-4490-947b-4bcc2ce9ee32-kube-api-access-x97z9" (OuterVolumeSpecName: "kube-api-access-x97z9") pod "bda8a039-b607-4490-947b-4bcc2ce9ee32" (UID: "bda8a039-b607-4490-947b-4bcc2ce9ee32"). InnerVolumeSpecName "kube-api-access-x97z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:34:04.737487 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.737447    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda8a039-b607-4490-947b-4bcc2ce9ee32-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:34:04.737487 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.737478    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97z9\" (UniqueName: \"kubernetes.io/projected/bda8a039-b607-4490-947b-4bcc2ce9ee32-kube-api-access-x97z9\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:34:04.737487 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:04.737490    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda8a039-b607-4490-947b-4bcc2ce9ee32-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:34:05.250113 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.250025    2584 generic.go:356] "Generic (PLEG): container finished" podID="bda8a039-b607-4490-947b-4bcc2ce9ee32" containerID="a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47" exitCode=0
Jul 14 02:34:05.250113 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.250073    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"
Jul 14 02:34:05.250304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.250110    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" event={"ID":"bda8a039-b607-4490-947b-4bcc2ce9ee32","Type":"ContainerDied","Data":"a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47"}
Jul 14 02:34:05.250304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.250180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt" event={"ID":"bda8a039-b607-4490-947b-4bcc2ce9ee32","Type":"ContainerDied","Data":"e73ebec67809c73218ddd6a67980c8b7f6cd27dd427c6fcbb214d23ec8a368ce"}
Jul 14 02:34:05.250304 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.250202    2584 scope.go:117] "RemoveContainer" containerID="a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47"
Jul 14 02:34:05.259108 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.259079    2584 scope.go:117] "RemoveContainer" containerID="a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47"
Jul 14 02:34:05.259383 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:34:05.259356    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47\": container with ID starting with a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47 not found: ID does not exist" containerID="a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47"
Jul 14 02:34:05.259453 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.259392    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47"} err="failed to get container status \"a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47\": rpc error: code = NotFound desc = could not find container \"a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47\": container with ID starting with a6afc9c194155b9369a582118ce1f0d2c1ab3f13d4e4a155ee84f172d36cea47 not found: ID does not exist"
Jul 14 02:34:05.265912 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.265887    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"]
Jul 14 02:34:05.267518 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:05.267498    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a5aba0cb-9c5dd74-sdxgt"]
Jul 14 02:34:06.891865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:34:06.891831    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda8a039-b607-4490-947b-4bcc2ce9ee32" path="/var/lib/kubelet/pods/bda8a039-b607-4490-947b-4bcc2ce9ee32/volumes"
Jul 14 02:35:57.015755 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.015724    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"]
Jul 14 02:35:57.016180 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.016045    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda8a039-b607-4490-947b-4bcc2ce9ee32" containerName="maas-api"
Jul 14 02:35:57.016180 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.016055    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda8a039-b607-4490-947b-4bcc2ce9ee32" containerName="maas-api"
Jul 14 02:35:57.016180 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.016123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda8a039-b607-4490-947b-4bcc2ce9ee32" containerName="maas-api"
Jul 14 02:35:57.018855 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.018836    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.021772 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.021753    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c40b4277-serving-cert\""
Jul 14 02:35:57.038574 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.038553    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"]
Jul 14 02:35:57.161359 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.161322    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlmr\" (UniqueName: \"kubernetes.io/projected/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-kube-api-access-jjlmr\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.161509 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.161381    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-openshift-service-ca\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.161509 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.161448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-maas-api-tls\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.262104 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.262072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlmr\" (UniqueName: \"kubernetes.io/projected/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-kube-api-access-jjlmr\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.262322 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.262120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-openshift-service-ca\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.262322 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.262179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-maas-api-tls\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.262828 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.262808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-openshift-service-ca\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.264510 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.264491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-maas-api-tls\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.270958 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.270909    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlmr\" (UniqueName: \"kubernetes.io/projected/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-kube-api-access-jjlmr\") pod \"maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.328357 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.328337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:57.463032 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.462995    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"]
Jul 14 02:35:57.466067 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:35:57.466039    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ed4680_4ddd_470b_9f5a_00c008fbb0f2.slice/crio-27c4b2fbbcc96e286efee9f577e6822a80619696496b99215e0f6c8464d0b14c WatchSource:0}: Error finding container 27c4b2fbbcc96e286efee9f577e6822a80619696496b99215e0f6c8464d0b14c: Status 404 returned error can't find the container with id 27c4b2fbbcc96e286efee9f577e6822a80619696496b99215e0f6c8464d0b14c
Jul 14 02:35:57.467381 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.467362    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 02:35:57.746270 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:57.746229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf" event={"ID":"53ed4680-4ddd-470b-9f5a-00c008fbb0f2","Type":"ContainerStarted","Data":"27c4b2fbbcc96e286efee9f577e6822a80619696496b99215e0f6c8464d0b14c"}
Jul 14 02:35:58.751993 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:58.751956    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf" event={"ID":"53ed4680-4ddd-470b-9f5a-00c008fbb0f2","Type":"ContainerStarted","Data":"989e463ded2fed76b845ba53fce0c924e300d3a0cbcaa8750db8c6bfe12e1741"}
Jul 14 02:35:58.752406 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:58.752084    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:35:58.768559 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:35:58.768515    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf" podStartSLOduration=2.238603687 podStartE2EDuration="2.768500899s" totalImagesPullingTime="529.897212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:35:58.767535052 +0000 UTC m=+2052.440409853" watchObservedRunningTime="2026-07-14 02:35:58.768500899 +0000 UTC m=+2052.441375699"
Jul 14 02:36:04.762903 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:04.762871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:36:18.714249 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:18.714155    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"]
Jul 14 02:36:18.714655 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:18.714390    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf" podUID="53ed4680-4ddd-470b-9f5a-00c008fbb0f2" containerName="maas-api" containerID="cri-o://989e463ded2fed76b845ba53fce0c924e300d3a0cbcaa8750db8c6bfe12e1741" gracePeriod=30
Jul 14 02:36:18.842620 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:18.842580    2584 generic.go:356] "Generic (PLEG): container finished" podID="53ed4680-4ddd-470b-9f5a-00c008fbb0f2" containerID="989e463ded2fed76b845ba53fce0c924e300d3a0cbcaa8750db8c6bfe12e1741" exitCode=0
Jul 14 02:36:18.842774 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:18.842650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf" event={"ID":"53ed4680-4ddd-470b-9f5a-00c008fbb0f2","Type":"ContainerDied","Data":"989e463ded2fed76b845ba53fce0c924e300d3a0cbcaa8750db8c6bfe12e1741"}
Jul 14 02:36:18.960441 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:18.960419    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:36:19.050419 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.050333    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-openshift-service-ca\") pod \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") "
Jul 14 02:36:19.050419 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.050397    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-maas-api-tls\") pod \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") "
Jul 14 02:36:19.050651 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.050429    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjlmr\" (UniqueName: \"kubernetes.io/projected/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-kube-api-access-jjlmr\") pod \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\" (UID: \"53ed4680-4ddd-470b-9f5a-00c008fbb0f2\") "
Jul 14 02:36:19.050712 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.050680    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53ed4680-4ddd-470b-9f5a-00c008fbb0f2" (UID: "53ed4680-4ddd-470b-9f5a-00c008fbb0f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:36:19.052554 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.052524    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-kube-api-access-jjlmr" (OuterVolumeSpecName: "kube-api-access-jjlmr") pod "53ed4680-4ddd-470b-9f5a-00c008fbb0f2" (UID: "53ed4680-4ddd-470b-9f5a-00c008fbb0f2"). InnerVolumeSpecName "kube-api-access-jjlmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:36:19.052554 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.052529    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53ed4680-4ddd-470b-9f5a-00c008fbb0f2" (UID: "53ed4680-4ddd-470b-9f5a-00c008fbb0f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:36:19.151305 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.151272    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:36:19.151305 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.151300    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:36:19.151305 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.151309    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlmr\" (UniqueName: \"kubernetes.io/projected/53ed4680-4ddd-470b-9f5a-00c008fbb0f2-kube-api-access-jjlmr\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:36:19.847936 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.847900    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf" event={"ID":"53ed4680-4ddd-470b-9f5a-00c008fbb0f2","Type":"ContainerDied","Data":"27c4b2fbbcc96e286efee9f577e6822a80619696496b99215e0f6c8464d0b14c"}
Jul 14 02:36:19.847936 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.847938    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"
Jul 14 02:36:19.848448 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.847985    2584 scope.go:117] "RemoveContainer" containerID="989e463ded2fed76b845ba53fce0c924e300d3a0cbcaa8750db8c6bfe12e1741"
Jul 14 02:36:19.867357 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.867334    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"]
Jul 14 02:36:19.870984 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:19.870963    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c40b4277-6f79df65d8-ltpvf"]
Jul 14 02:36:20.890994 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:20.890960    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ed4680-4ddd-470b-9f5a-00c008fbb0f2" path="/var/lib/kubelet/pods/53ed4680-4ddd-470b-9f5a-00c008fbb0f2/volumes"
Jul 14 02:36:56.174110 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.174081    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"]
Jul 14 02:36:56.174499 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.174418    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ed4680-4ddd-470b-9f5a-00c008fbb0f2" containerName="maas-api"
Jul 14 02:36:56.174499 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.174429    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ed4680-4ddd-470b-9f5a-00c008fbb0f2" containerName="maas-api"
Jul 14 02:36:56.174499 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.174497    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ed4680-4ddd-470b-9f5a-00c008fbb0f2" containerName="maas-api"
Jul 14 02:36:56.177404 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.177386    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.179840 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.179813    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ca6f969a-serving-cert\""
Jul 14 02:36:56.186511 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.186489    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"]
Jul 14 02:36:56.232596 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.232565    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-maas-api-tls\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.232596 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.232596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b74z\" (UniqueName: \"kubernetes.io/projected/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-kube-api-access-4b74z\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.232782 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.232637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-openshift-service-ca\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.333808 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.333769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-openshift-service-ca\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.333973 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.333828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-maas-api-tls\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.333973 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.333846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b74z\" (UniqueName: \"kubernetes.io/projected/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-kube-api-access-4b74z\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.334473 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.334450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-openshift-service-ca\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.336300 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.336284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-maas-api-tls\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.341116 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.341090    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b74z\" (UniqueName: \"kubernetes.io/projected/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-kube-api-access-4b74z\") pod \"maas-api-e2e-mt-ca6f969a-857996684-brljd\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.489956 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.489868    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:56.618687 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:56.618661    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"]
Jul 14 02:36:56.621359 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:36:56.621331    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e37cc3_deb2_4eb4_9b23_55bd8ffce99a.slice/crio-5029b42ef94a23232b651dbd4e5291126dd964a0dfdf64c8d3923a3b974c6cb9 WatchSource:0}: Error finding container 5029b42ef94a23232b651dbd4e5291126dd964a0dfdf64c8d3923a3b974c6cb9: Status 404 returned error can't find the container with id 5029b42ef94a23232b651dbd4e5291126dd964a0dfdf64c8d3923a3b974c6cb9
Jul 14 02:36:57.009577 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:57.009546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd" event={"ID":"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a","Type":"ContainerStarted","Data":"128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b"}
Jul 14 02:36:57.009730 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:57.009595    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd" event={"ID":"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a","Type":"ContainerStarted","Data":"5029b42ef94a23232b651dbd4e5291126dd964a0dfdf64c8d3923a3b974c6cb9"}
Jul 14 02:36:57.009730 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:57.009624    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:36:57.026358 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:36:57.026310    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd" podStartSLOduration=0.726395516 podStartE2EDuration="1.026294655s" totalImagesPullingTime="299.899139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:36:57.024517722 +0000 UTC m=+2110.697392524" watchObservedRunningTime="2026-07-14 02:36:57.026294655 +0000 UTC m=+2110.699169455"
Jul 14 02:37:03.019722 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:03.019688    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:37:49.295040 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.294962    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"]
Jul 14 02:37:49.295597 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.295216    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd" podUID="64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" containerName="maas-api" containerID="cri-o://128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b" gracePeriod=30
Jul 14 02:37:49.539534 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.539511    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:37:49.587539 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.587470    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-openshift-service-ca\") pod \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") "
Jul 14 02:37:49.587539 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.587525    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-maas-api-tls\") pod \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") "
Jul 14 02:37:49.587795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.587771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" (UID: "64e37cc3-deb2-4eb4-9b23-55bd8ffce99a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:37:49.589465 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.589437    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" (UID: "64e37cc3-deb2-4eb4-9b23-55bd8ffce99a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:37:49.688629 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.688600    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b74z\" (UniqueName: \"kubernetes.io/projected/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-kube-api-access-4b74z\") pod \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\" (UID: \"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a\") "
Jul 14 02:37:49.688763 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.688736    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:37:49.688763 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.688747    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:37:49.690571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.690545    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-kube-api-access-4b74z" (OuterVolumeSpecName: "kube-api-access-4b74z") pod "64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" (UID: "64e37cc3-deb2-4eb4-9b23-55bd8ffce99a"). InnerVolumeSpecName "kube-api-access-4b74z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:37:49.789927 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:49.789899    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b74z\" (UniqueName: \"kubernetes.io/projected/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a-kube-api-access-4b74z\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:37:50.240627 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.240589    2584 generic.go:356] "Generic (PLEG): container finished" podID="64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" containerID="128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b" exitCode=0
Jul 14 02:37:50.240793 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.240637    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"
Jul 14 02:37:50.240793 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.240644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd" event={"ID":"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a","Type":"ContainerDied","Data":"128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b"}
Jul 14 02:37:50.240793 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.240695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd" event={"ID":"64e37cc3-deb2-4eb4-9b23-55bd8ffce99a","Type":"ContainerDied","Data":"5029b42ef94a23232b651dbd4e5291126dd964a0dfdf64c8d3923a3b974c6cb9"}
Jul 14 02:37:50.240793 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.240726    2584 scope.go:117] "RemoveContainer" containerID="128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b"
Jul 14 02:37:50.249171 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.249154    2584 scope.go:117] "RemoveContainer" containerID="128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b"
Jul 14 02:37:50.249443 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:37:50.249425    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b\": container with ID starting with 128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b not found: ID does not exist" containerID="128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b"
Jul 14 02:37:50.249490 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.249454    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b"} err="failed to get container status \"128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b\": rpc error: code = NotFound desc = could not find container \"128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b\": container with ID starting with 128c5644af02ad24cb78ff677cdb86be0eb1de7154701e0d412b44ad3804047b not found: ID does not exist"
Jul 14 02:37:50.261686 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.261662    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"]
Jul 14 02:37:50.263453 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.263433    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6f969a-857996684-brljd"]
Jul 14 02:37:50.892391 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:37:50.892357    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" path="/var/lib/kubelet/pods/64e37cc3-deb2-4eb4-9b23-55bd8ffce99a/volumes"
Jul 14 02:39:43.040266 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.040165    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"]
Jul 14 02:39:43.042620 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.040543    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" containerName="maas-api"
Jul 14 02:39:43.042620 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.040559    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" containerName="maas-api"
Jul 14 02:39:43.042620 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.040635    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e37cc3-deb2-4eb4-9b23-55bd8ffce99a" containerName="maas-api"
Jul 14 02:39:43.043506 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.043488    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.045962 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.045939    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cc2dba-serving-cert\""
Jul 14 02:39:43.053476 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.053454    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"]
Jul 14 02:39:43.141055 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.141014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqpm\" (UniqueName: \"kubernetes.io/projected/c39d5769-a319-4fec-b38b-50ad9a3c1187-kube-api-access-zrqpm\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.141055 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.141060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d5769-a319-4fec-b38b-50ad9a3c1187-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.141278 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.141150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d5769-a319-4fec-b38b-50ad9a3c1187-maas-api-tls\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.241541 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.241506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d5769-a319-4fec-b38b-50ad9a3c1187-maas-api-tls\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.241739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.241594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqpm\" (UniqueName: \"kubernetes.io/projected/c39d5769-a319-4fec-b38b-50ad9a3c1187-kube-api-access-zrqpm\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.241739 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.241620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d5769-a319-4fec-b38b-50ad9a3c1187-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.242263 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.242237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d5769-a319-4fec-b38b-50ad9a3c1187-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.244199 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.244035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d5769-a319-4fec-b38b-50ad9a3c1187-maas-api-tls\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.249199 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.249176    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqpm\" (UniqueName: \"kubernetes.io/projected/c39d5769-a319-4fec-b38b-50ad9a3c1187-kube-api-access-zrqpm\") pod \"maas-api-e2e-inf-a-cc2dba-654f45976b-kw847\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.355725 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.355645    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:43.489068 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.489025    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"]
Jul 14 02:39:43.491160 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:39:43.491120    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39d5769_a319_4fec_b38b_50ad9a3c1187.slice/crio-01961d1ed4bc37b0e867fa7fe6e5ac226621cf6fd84a4edadc2ef6a656cf5c53 WatchSource:0}: Error finding container 01961d1ed4bc37b0e867fa7fe6e5ac226621cf6fd84a4edadc2ef6a656cf5c53: Status 404 returned error can't find the container with id 01961d1ed4bc37b0e867fa7fe6e5ac226621cf6fd84a4edadc2ef6a656cf5c53
Jul 14 02:39:43.738363 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:43.738330    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847" event={"ID":"c39d5769-a319-4fec-b38b-50ad9a3c1187","Type":"ContainerStarted","Data":"01961d1ed4bc37b0e867fa7fe6e5ac226621cf6fd84a4edadc2ef6a656cf5c53"}
Jul 14 02:39:44.744569 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:44.744533    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847" event={"ID":"c39d5769-a319-4fec-b38b-50ad9a3c1187","Type":"ContainerStarted","Data":"b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8"}
Jul 14 02:39:44.744965 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:44.744625    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:44.760193 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:44.760149    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847" podStartSLOduration=1.299538845 podStartE2EDuration="1.76011869s" totalImagesPullingTime="460.579845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:39:44.758531056 +0000 UTC m=+2278.431405859" watchObservedRunningTime="2026-07-14 02:39:44.76011869 +0000 UTC m=+2278.432993490"
Jul 14 02:39:50.754861 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:50.754827    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:39:51.070159 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.070056    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"]
Jul 14 02:39:51.073985 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.073961    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.076421 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.076397    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7924bc-serving-cert\""
Jul 14 02:39:51.085834 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.085804    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"]
Jul 14 02:39:51.112669 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.112640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c601be06-17bd-4697-8a2d-bb5d4afc3c55-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.112801 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.112673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c601be06-17bd-4697-8a2d-bb5d4afc3c55-maas-api-tls\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.112801 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.112696    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtp9\" (UniqueName: \"kubernetes.io/projected/c601be06-17bd-4697-8a2d-bb5d4afc3c55-kube-api-access-jmtp9\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.213439 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.213411    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c601be06-17bd-4697-8a2d-bb5d4afc3c55-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.213439 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.213443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c601be06-17bd-4697-8a2d-bb5d4afc3c55-maas-api-tls\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.213693 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.213471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtp9\" (UniqueName: \"kubernetes.io/projected/c601be06-17bd-4697-8a2d-bb5d4afc3c55-kube-api-access-jmtp9\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.214117 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.214093    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c601be06-17bd-4697-8a2d-bb5d4afc3c55-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.215829 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.215810    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c601be06-17bd-4697-8a2d-bb5d4afc3c55-maas-api-tls\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.223977 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.223950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtp9\" (UniqueName: \"kubernetes.io/projected/c601be06-17bd-4697-8a2d-bb5d4afc3c55-kube-api-access-jmtp9\") pod \"maas-api-e2e-inf-b-7924bc-c9458585-nrrks\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.385756 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.385731    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:51.518108 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.518051    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"]
Jul 14 02:39:51.521502 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:39:51.521473    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc601be06_17bd_4697_8a2d_bb5d4afc3c55.slice/crio-9a6ba0063896cd21db2ce43fbf7c13b2497acb7c5e26d52d8e3b7e53def81878 WatchSource:0}: Error finding container 9a6ba0063896cd21db2ce43fbf7c13b2497acb7c5e26d52d8e3b7e53def81878: Status 404 returned error can't find the container with id 9a6ba0063896cd21db2ce43fbf7c13b2497acb7c5e26d52d8e3b7e53def81878
Jul 14 02:39:51.776099 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:51.776015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks" event={"ID":"c601be06-17bd-4697-8a2d-bb5d4afc3c55","Type":"ContainerStarted","Data":"9a6ba0063896cd21db2ce43fbf7c13b2497acb7c5e26d52d8e3b7e53def81878"}
Jul 14 02:39:52.782554 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:52.782517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks" event={"ID":"c601be06-17bd-4697-8a2d-bb5d4afc3c55","Type":"ContainerStarted","Data":"6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6"}
Jul 14 02:39:52.782932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:52.782664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:39:52.798806 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:52.798761    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks" podStartSLOduration=1.21149146 podStartE2EDuration="1.798747238s" totalImagesPullingTime="587.255778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:39:52.797630821 +0000 UTC m=+2286.470505624" watchObservedRunningTime="2026-07-14 02:39:52.798747238 +0000 UTC m=+2286.471622037"
Jul 14 02:39:58.792621 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:39:58.792590    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:40:29.495292 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.495262    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"]
Jul 14 02:40:29.498810 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.498793    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.501228 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.501208    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7924bc\"/\"kube-root-ca.crt\""
Jul 14 02:40:29.501342 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.501239    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7924bc\"/\"default-dockercfg-tzcbc\""
Jul 14 02:40:29.502367 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.502340    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7924bc\"/\"test-model-7924bc-kserve-self-signed-certs\""
Jul 14 02:40:29.502456 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.502385    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7924bc\"/\"openshift-service-ca.crt\""
Jul 14 02:40:29.505968 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.505942    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"]
Jul 14 02:40:29.545070 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.545023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tls-certs\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.545265 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.545077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-dshm\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.545265 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.545191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-home\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.545265 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.545217    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9r8\" (UniqueName: \"kubernetes.io/projected/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-kube-api-access-bh9r8\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.545265 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.545250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-model-cache\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.545265 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.545265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tmp-dir\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.646501 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.646449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tls-certs\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.646501 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.646509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-dshm\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.646736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.646563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-home\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.646736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.646583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9r8\" (UniqueName: \"kubernetes.io/projected/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-kube-api-access-bh9r8\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.646736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.646604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-model-cache\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.646736 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.646626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tmp-dir\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.647059 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.647028    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-model-cache\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.647221 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.647046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tmp-dir\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.647221 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.647118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-home\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.648848 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.648826    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-dshm\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.649104 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.649085    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tls-certs\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.654182 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.654161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9r8\" (UniqueName: \"kubernetes.io/projected/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-kube-api-access-bh9r8\") pod \"test-model-7924bc-kserve-67745c7bc-x6qrn\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") " pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.809351 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.809261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:29.937531 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.937506    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"]
Jul 14 02:40:29.939030 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:40:29.938998    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b5ee6e3_b2e9_42c4_ad18_5db9f3c28127.slice/crio-7ebc51093530b31ebb4cf42197e9a5a0a33f3315929220091c3429f0eed4612d WatchSource:0}: Error finding container 7ebc51093530b31ebb4cf42197e9a5a0a33f3315929220091c3429f0eed4612d: Status 404 returned error can't find the container with id 7ebc51093530b31ebb4cf42197e9a5a0a33f3315929220091c3429f0eed4612d
Jul 14 02:40:29.957028 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:29.957001    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn" event={"ID":"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127","Type":"ContainerStarted","Data":"7ebc51093530b31ebb4cf42197e9a5a0a33f3315929220091c3429f0eed4612d"}
Jul 14 02:40:30.969865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:30.969830    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn" event={"ID":"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127","Type":"ContainerStarted","Data":"c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551"}
Jul 14 02:40:30.987866 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:30.987820    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn" podStartSLOduration=1.9878059609999998 podStartE2EDuration="1.987805961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:40:30.986744708 +0000 UTC m=+2324.659619510" watchObservedRunningTime="2026-07-14 02:40:30.987805961 +0000 UTC m=+2324.660680801"
Jul 14 02:40:39.810354 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:39.810263    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:39.810354 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:39.810309    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:39.823309 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:39.823277    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:40.022705 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:40.022674    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:40:49.822662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:40:49.822625    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"]
Jul 14 02:41:00.270450 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.270414    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"]
Jul 14 02:41:00.270887 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.270651    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847" podUID="c39d5769-a319-4fec-b38b-50ad9a3c1187" containerName="maas-api" containerID="cri-o://b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8" gracePeriod=30
Jul 14 02:41:00.716226 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.716200    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:41:00.814356 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.814284    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrqpm\" (UniqueName: \"kubernetes.io/projected/c39d5769-a319-4fec-b38b-50ad9a3c1187-kube-api-access-zrqpm\") pod \"c39d5769-a319-4fec-b38b-50ad9a3c1187\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") "
Jul 14 02:41:00.814495 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.814372    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d5769-a319-4fec-b38b-50ad9a3c1187-openshift-service-ca\") pod \"c39d5769-a319-4fec-b38b-50ad9a3c1187\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") "
Jul 14 02:41:00.814495 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.814404    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d5769-a319-4fec-b38b-50ad9a3c1187-maas-api-tls\") pod \"c39d5769-a319-4fec-b38b-50ad9a3c1187\" (UID: \"c39d5769-a319-4fec-b38b-50ad9a3c1187\") "
Jul 14 02:41:00.814716 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.814690    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39d5769-a319-4fec-b38b-50ad9a3c1187-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c39d5769-a319-4fec-b38b-50ad9a3c1187" (UID: "c39d5769-a319-4fec-b38b-50ad9a3c1187"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:41:00.816389 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.816354    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39d5769-a319-4fec-b38b-50ad9a3c1187-kube-api-access-zrqpm" (OuterVolumeSpecName: "kube-api-access-zrqpm") pod "c39d5769-a319-4fec-b38b-50ad9a3c1187" (UID: "c39d5769-a319-4fec-b38b-50ad9a3c1187"). InnerVolumeSpecName "kube-api-access-zrqpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:41:00.816488 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.816396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39d5769-a319-4fec-b38b-50ad9a3c1187-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c39d5769-a319-4fec-b38b-50ad9a3c1187" (UID: "c39d5769-a319-4fec-b38b-50ad9a3c1187"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:41:00.915588 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.915562    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqpm\" (UniqueName: \"kubernetes.io/projected/c39d5769-a319-4fec-b38b-50ad9a3c1187-kube-api-access-zrqpm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:00.915588 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.915585    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d5769-a319-4fec-b38b-50ad9a3c1187-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:00.915735 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:00.915594    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d5769-a319-4fec-b38b-50ad9a3c1187-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:01.105754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.105668    2584 generic.go:356] "Generic (PLEG): container finished" podID="c39d5769-a319-4fec-b38b-50ad9a3c1187" containerID="b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8" exitCode=0
Jul 14 02:41:01.105754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.105714    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"
Jul 14 02:41:01.105754 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.105738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847" event={"ID":"c39d5769-a319-4fec-b38b-50ad9a3c1187","Type":"ContainerDied","Data":"b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8"}
Jul 14 02:41:01.106022 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.105784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847" event={"ID":"c39d5769-a319-4fec-b38b-50ad9a3c1187","Type":"ContainerDied","Data":"01961d1ed4bc37b0e867fa7fe6e5ac226621cf6fd84a4edadc2ef6a656cf5c53"}
Jul 14 02:41:01.106022 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.105807    2584 scope.go:117] "RemoveContainer" containerID="b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8"
Jul 14 02:41:01.114762 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.114739    2584 scope.go:117] "RemoveContainer" containerID="b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8"
Jul 14 02:41:01.115033 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:01.115012    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8\": container with ID starting with b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8 not found: ID does not exist" containerID="b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8"
Jul 14 02:41:01.115092 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.115041    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8"} err="failed to get container status \"b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8\": rpc error: code = NotFound desc = could not find container \"b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8\": container with ID starting with b9cd4892a7b312eba45d80c33b6cfe87f3676324cfcba57a72b7b02960f38ff8 not found: ID does not exist"
Jul 14 02:41:01.121515 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.121489    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"]
Jul 14 02:41:01.126719 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:01.126698    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc2dba-654f45976b-kw847"]
Jul 14 02:41:02.891388 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:02.891355    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c39d5769-a319-4fec-b38b-50ad9a3c1187" path="/var/lib/kubelet/pods/c39d5769-a319-4fec-b38b-50ad9a3c1187/volumes"
Jul 14 02:41:04.843542 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:04.843484    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn" podUID="7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" containerName="main" containerID="cri-o://c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551" gracePeriod=45
Jul 14 02:41:05.088802 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.088780    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:41:05.125185 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.125079    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" containerID="c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551" exitCode=0
Jul 14 02:41:05.125185 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.125142    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"
Jul 14 02:41:05.125185 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.125168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn" event={"ID":"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127","Type":"ContainerDied","Data":"c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551"}
Jul 14 02:41:05.125433 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.125210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn" event={"ID":"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127","Type":"ContainerDied","Data":"7ebc51093530b31ebb4cf42197e9a5a0a33f3315929220091c3429f0eed4612d"}
Jul 14 02:41:05.125433 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.125227    2584 scope.go:117] "RemoveContainer" containerID="c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551"
Jul 14 02:41:05.134894 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.134875    2584 scope.go:117] "RemoveContainer" containerID="c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551"
Jul 14 02:41:05.135259 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:05.135240    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551\": container with ID starting with c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551 not found: ID does not exist" containerID="c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551"
Jul 14 02:41:05.135317 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.135268    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551"} err="failed to get container status \"c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551\": rpc error: code = NotFound desc = could not find container \"c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551\": container with ID starting with c80c98edc81e27c665d72fcf20550b2433ba2b913fb9c366dc05c89263dce551 not found: ID does not exist"
Jul 14 02:41:05.250417 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250388    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-dshm\") pod \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") "
Jul 14 02:41:05.250591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250440    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tls-certs\") pod \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") "
Jul 14 02:41:05.250591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250459    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-model-cache\") pod \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") "
Jul 14 02:41:05.250591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250484    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tmp-dir\") pod \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") "
Jul 14 02:41:05.250591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250525    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-home\") pod \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") "
Jul 14 02:41:05.250591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250549    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh9r8\" (UniqueName: \"kubernetes.io/projected/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-kube-api-access-bh9r8\") pod \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\" (UID: \"7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127\") "
Jul 14 02:41:05.250865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250790    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" (UID: "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:41:05.250865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250803    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-home" (OuterVolumeSpecName: "home") pod "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" (UID: "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:41:05.250865 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250816    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" (UID: "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:41:05.251000 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250900    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:05.251000 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250918    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:05.251000 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.250931    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:05.252624 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.252601    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" (UID: "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:41:05.252907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.252889    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-dshm" (OuterVolumeSpecName: "dshm") pod "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" (UID: "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:41:05.252907 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.252896    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-kube-api-access-bh9r8" (OuterVolumeSpecName: "kube-api-access-bh9r8") pod "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" (UID: "7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127"). InnerVolumeSpecName "kube-api-access-bh9r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:41:05.351522 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.351476    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:05.351522 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.351526    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:05.351522 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.351538    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9r8\" (UniqueName: \"kubernetes.io/projected/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127-kube-api-access-bh9r8\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:05.445879 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.445848    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"]
Jul 14 02:41:05.451258 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:05.451235    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7924bc/test-model-7924bc-kserve-67745c7bc-x6qrn"]
Jul 14 02:41:06.892183 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:06.892122    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" path="/var/lib/kubelet/pods/7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127/volumes"
Jul 14 02:41:26.394414 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.394371    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"]
Jul 14 02:41:26.394933 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.394901    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks" podUID="c601be06-17bd-4697-8a2d-bb5d4afc3c55" containerName="maas-api" containerID="cri-o://6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6" gracePeriod=30
Jul 14 02:41:26.646632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.646575    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:41:26.739210 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.739180    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmtp9\" (UniqueName: \"kubernetes.io/projected/c601be06-17bd-4697-8a2d-bb5d4afc3c55-kube-api-access-jmtp9\") pod \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") "
Jul 14 02:41:26.739210 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.739212    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c601be06-17bd-4697-8a2d-bb5d4afc3c55-maas-api-tls\") pod \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") "
Jul 14 02:41:26.739416 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.739378    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c601be06-17bd-4697-8a2d-bb5d4afc3c55-openshift-service-ca\") pod \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\" (UID: \"c601be06-17bd-4697-8a2d-bb5d4afc3c55\") "
Jul 14 02:41:26.739720 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.739691    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c601be06-17bd-4697-8a2d-bb5d4afc3c55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c601be06-17bd-4697-8a2d-bb5d4afc3c55" (UID: "c601be06-17bd-4697-8a2d-bb5d4afc3c55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:41:26.741259 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.741236    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c601be06-17bd-4697-8a2d-bb5d4afc3c55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c601be06-17bd-4697-8a2d-bb5d4afc3c55" (UID: "c601be06-17bd-4697-8a2d-bb5d4afc3c55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:41:26.741365 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.741319    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c601be06-17bd-4697-8a2d-bb5d4afc3c55-kube-api-access-jmtp9" (OuterVolumeSpecName: "kube-api-access-jmtp9") pod "c601be06-17bd-4697-8a2d-bb5d4afc3c55" (UID: "c601be06-17bd-4697-8a2d-bb5d4afc3c55"). InnerVolumeSpecName "kube-api-access-jmtp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:41:26.840407 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.840375    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmtp9\" (UniqueName: \"kubernetes.io/projected/c601be06-17bd-4697-8a2d-bb5d4afc3c55-kube-api-access-jmtp9\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:26.840407 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.840399    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c601be06-17bd-4697-8a2d-bb5d4afc3c55-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:26.840407 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:26.840409    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c601be06-17bd-4697-8a2d-bb5d4afc3c55-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:41:27.224413 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.224326    2584 generic.go:356] "Generic (PLEG): container finished" podID="c601be06-17bd-4697-8a2d-bb5d4afc3c55" containerID="6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6" exitCode=0
Jul 14 02:41:27.224413 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.224368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks" event={"ID":"c601be06-17bd-4697-8a2d-bb5d4afc3c55","Type":"ContainerDied","Data":"6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6"}
Jul 14 02:41:27.224413 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.224369    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"
Jul 14 02:41:27.224413 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.224412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks" event={"ID":"c601be06-17bd-4697-8a2d-bb5d4afc3c55","Type":"ContainerDied","Data":"9a6ba0063896cd21db2ce43fbf7c13b2497acb7c5e26d52d8e3b7e53def81878"}
Jul 14 02:41:27.224666 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.224436    2584 scope.go:117] "RemoveContainer" containerID="6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6"
Jul 14 02:41:27.232882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.232866    2584 scope.go:117] "RemoveContainer" containerID="6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6"
Jul 14 02:41:27.233119 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:27.233098    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6\": container with ID starting with 6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6 not found: ID does not exist" containerID="6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6"
Jul 14 02:41:27.233189 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.233141    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6"} err="failed to get container status \"6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6\": rpc error: code = NotFound desc = could not find container \"6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6\": container with ID starting with 6ff9680ce12e6ba4a74a5bfbe8493818ef93124ad3f6d1f57f5eaa655da31eb6 not found: ID does not exist"
Jul 14 02:41:27.241849 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.241818    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"]
Jul 14 02:41:27.243903 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:27.243882    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7924bc-c9458585-nrrks"]
Jul 14 02:41:28.892437 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:28.892403    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c601be06-17bd-4697-8a2d-bb5d4afc3c55" path="/var/lib/kubelet/pods/c601be06-17bd-4697-8a2d-bb5d4afc3c55/volumes"
Jul 14 02:41:45.662628 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.662588    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"]
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.662968    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c39d5769-a319-4fec-b38b-50ad9a3c1187" containerName="maas-api"
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.662981    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c39d5769-a319-4fec-b38b-50ad9a3c1187" containerName="maas-api"
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.662994    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" containerName="main"
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.663000    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" containerName="main"
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.663019    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c601be06-17bd-4697-8a2d-bb5d4afc3c55" containerName="maas-api"
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.663025    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c601be06-17bd-4697-8a2d-bb5d4afc3c55" containerName="maas-api"
Jul 14 02:41:45.663106 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.663099    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c39d5769-a319-4fec-b38b-50ad9a3c1187" containerName="maas-api"
Jul 14 02:41:45.663479 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.663114    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c601be06-17bd-4697-8a2d-bb5d4afc3c55" containerName="maas-api"
Jul 14 02:41:45.663479 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.663142    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5ee6e3-b2e9-42c4-ad18-5db9f3c28127" containerName="main"
Jul 14 02:41:45.666196 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.666180    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.669451 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.669428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9794b6-serving-cert\""
Jul 14 02:41:45.698938 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.698907    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"]
Jul 14 02:41:45.805272 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.805237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.805445 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.805278    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-openshift-service-ca\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.805445 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.805300    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqplm\" (UniqueName: \"kubernetes.io/projected/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-kube-api-access-kqplm\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.906650 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.906615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.906650 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.906654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-openshift-service-ca\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.906850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.906676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqplm\" (UniqueName: \"kubernetes.io/projected/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-kube-api-access-kqplm\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.906850 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:45.906769    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-serving-cert: secret "maas-api-e2e-api-a-9794b6-serving-cert" not found
Jul 14 02:41:45.906850 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:45.906833    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls podName:06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd nodeName:}" failed. No retries permitted until 2026-07-14 02:41:46.406817482 +0000 UTC m=+2400.079692263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls") pod "maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" (UID: "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd") : secret "maas-api-e2e-api-a-9794b6-serving-cert" not found
Jul 14 02:41:45.907399 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.907379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-openshift-service-ca\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:45.918043 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:45.917979    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqplm\" (UniqueName: \"kubernetes.io/projected/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-kube-api-access-kqplm\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:46.410668 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:46.410632    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:46.413171 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:46.413125    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls\") pod \"maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:46.576862 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:46.576832    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:46.718592 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:46.718568    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"]
Jul 14 02:41:46.720807 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:41:46.720780    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cc7c65_8fe8_4f19_88ec_2e5b51c6d1dd.slice/crio-752c40903c084616d8a6a711f421035c5799211e0ddc0c4fec7a06cd45d4245d WatchSource:0}: Error finding container 752c40903c084616d8a6a711f421035c5799211e0ddc0c4fec7a06cd45d4245d: Status 404 returned error can't find the container with id 752c40903c084616d8a6a711f421035c5799211e0ddc0c4fec7a06cd45d4245d
Jul 14 02:41:46.722494 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:46.722475    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 02:41:47.316583 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:47.316495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" event={"ID":"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd","Type":"ContainerStarted","Data":"752c40903c084616d8a6a711f421035c5799211e0ddc0c4fec7a06cd45d4245d"}
Jul 14 02:41:48.321856 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:48.321816    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" event={"ID":"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd","Type":"ContainerStarted","Data":"8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb"}
Jul 14 02:41:48.322309 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:48.321951    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:48.342187 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:48.342118    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" podStartSLOduration=2.669344348 podStartE2EDuration="3.342104086s" totalImagesPullingTime="672.759738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:41:48.336640833 +0000 UTC m=+2402.009515633" watchObservedRunningTime="2026-07-14 02:41:48.342104086 +0000 UTC m=+2402.014978885"
Jul 14 02:41:53.726150 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.726072    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"]
Jul 14 02:41:53.729443 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.729421    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.731869 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.731848    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f83e72-serving-cert\""
Jul 14 02:41:53.740511 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.740484    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"]
Jul 14 02:41:53.871664 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.871627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-openshift-service-ca\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.871840 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.871681    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.871840 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.871757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx42v\" (UniqueName: \"kubernetes.io/projected/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-kube-api-access-tx42v\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.972612 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.972579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx42v\" (UniqueName: \"kubernetes.io/projected/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-kube-api-access-tx42v\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.972799 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.972640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-openshift-service-ca\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.972799 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.972753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.972915 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:53.972873    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-serving-cert: secret "maas-api-e2e-api-b-f83e72-serving-cert" not found
Jul 14 02:41:53.972971 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:41:53.972931    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls podName:eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737 nodeName:}" failed. No retries permitted until 2026-07-14 02:41:54.47291583 +0000 UTC m=+2408.145790611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls") pod "maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" (UID: "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737") : secret "maas-api-e2e-api-b-f83e72-serving-cert" not found
Jul 14 02:41:53.973258 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.973241    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-openshift-service-ca\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:53.981653 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:53.981602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx42v\" (UniqueName: \"kubernetes.io/projected/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-kube-api-access-tx42v\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:54.331325 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:54.331241    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:41:54.477316 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:54.477275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:54.480086 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:54.480063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls\") pod \"maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:54.641748 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:54.641714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:54.772832 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:54.772808    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"]
Jul 14 02:41:54.775071 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:41:54.775034    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9199eb_ad2e_4f3d_92ed_45fe3b0e8737.slice/crio-227615f15fcda71729682e9c922ccc2664810247a960a294e79f6a38e09dedae WatchSource:0}: Error finding container 227615f15fcda71729682e9c922ccc2664810247a960a294e79f6a38e09dedae: Status 404 returned error can't find the container with id 227615f15fcda71729682e9c922ccc2664810247a960a294e79f6a38e09dedae
Jul 14 02:41:55.355600 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:55.355556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" event={"ID":"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737","Type":"ContainerStarted","Data":"8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7"}
Jul 14 02:41:55.355776 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:55.355612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" event={"ID":"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737","Type":"ContainerStarted","Data":"227615f15fcda71729682e9c922ccc2664810247a960a294e79f6a38e09dedae"}
Jul 14 02:41:55.355776 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:55.355703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:41:55.372562 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:41:55.372518    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" podStartSLOduration=1.993782681 podStartE2EDuration="2.372504245s" totalImagesPullingTime="378.721564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:41:55.371529593 +0000 UTC m=+2409.044404393" watchObservedRunningTime="2026-07-14 02:41:55.372504245 +0000 UTC m=+2409.045379070"
Jul 14 02:42:01.365590 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:01.365560    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:42:16.802192 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:16.802092    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"]
Jul 14 02:42:16.802659 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:16.802367    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" podUID="06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" containerName="maas-api" containerID="cri-o://8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb" gracePeriod=30
Jul 14 02:42:17.056745 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.056686    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:42:17.076744 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.076699    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqplm\" (UniqueName: \"kubernetes.io/projected/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-kube-api-access-kqplm\") pod \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") "
Jul 14 02:42:17.076913 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.076784    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls\") pod \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") "
Jul 14 02:42:17.076913 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.076823    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-openshift-service-ca\") pod \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\" (UID: \"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd\") "
Jul 14 02:42:17.077420 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.077391    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" (UID: "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:42:17.078838 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.078814    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-kube-api-access-kqplm" (OuterVolumeSpecName: "kube-api-access-kqplm") pod "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" (UID: "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd"). InnerVolumeSpecName "kube-api-access-kqplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:42:17.079291 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.079247    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" (UID: "06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:42:17.178471 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.178436    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqplm\" (UniqueName: \"kubernetes.io/projected/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-kube-api-access-kqplm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:42:17.178471 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.178464    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:42:17.178471 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.178473    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:42:17.464966 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.464929    2584 generic.go:356] "Generic (PLEG): container finished" podID="06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" containerID="8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb" exitCode=0
Jul 14 02:42:17.465174 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.464984    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"
Jul 14 02:42:17.465174 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.465019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" event={"ID":"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd","Type":"ContainerDied","Data":"8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb"}
Jul 14 02:42:17.465174 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.465099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw" event={"ID":"06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd","Type":"ContainerDied","Data":"752c40903c084616d8a6a711f421035c5799211e0ddc0c4fec7a06cd45d4245d"}
Jul 14 02:42:17.465174 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.465125    2584 scope.go:117] "RemoveContainer" containerID="8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb"
Jul 14 02:42:17.474167 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.474147    2584 scope.go:117] "RemoveContainer" containerID="8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb"
Jul 14 02:42:17.474425 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:42:17.474408    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb\": container with ID starting with 8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb not found: ID does not exist" containerID="8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb"
Jul 14 02:42:17.474494 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.474431    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb"} err="failed to get container status \"8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb\": rpc error: code = NotFound desc = could not find container \"8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb\": container with ID starting with 8c63b2fdbf879a0d281652e62a3917829b704bd5d82db9fcfc2f75ee86f27fdb not found: ID does not exist"
Jul 14 02:42:17.485415 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.485385    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"]
Jul 14 02:42:17.487230 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:17.487211    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9794b6-69b9ccfbc9-t22tw"]
Jul 14 02:42:18.891388 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:18.891356    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" path="/var/lib/kubelet/pods/06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd/volumes"
Jul 14 02:42:42.574793 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.574759    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"]
Jul 14 02:42:42.575686 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.575650    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" podUID="eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" containerName="maas-api" containerID="cri-o://8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7" gracePeriod=30
Jul 14 02:42:42.829107 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.829051    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:42:42.903686 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.903660    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-openshift-service-ca\") pod \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") "
Jul 14 02:42:42.903856 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.903722    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls\") pod \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") "
Jul 14 02:42:42.903856 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.903823    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx42v\" (UniqueName: \"kubernetes.io/projected/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-kube-api-access-tx42v\") pod \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\" (UID: \"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737\") "
Jul 14 02:42:42.904029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.904005    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" (UID: "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:42:42.905700 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.905675    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" (UID: "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:42:42.905784 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:42.905732    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-kube-api-access-tx42v" (OuterVolumeSpecName: "kube-api-access-tx42v") pod "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" (UID: "eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737"). InnerVolumeSpecName "kube-api-access-tx42v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:42:43.005033 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.004998    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx42v\" (UniqueName: \"kubernetes.io/projected/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-kube-api-access-tx42v\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:42:43.005033 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.005029    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:42:43.005270 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.005045    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:42:43.584202 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.584172    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" containerID="8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7" exitCode=0
Jul 14 02:42:43.584701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.584224    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"
Jul 14 02:42:43.584701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.584277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" event={"ID":"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737","Type":"ContainerDied","Data":"8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7"}
Jul 14 02:42:43.584701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.584341    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x" event={"ID":"eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737","Type":"ContainerDied","Data":"227615f15fcda71729682e9c922ccc2664810247a960a294e79f6a38e09dedae"}
Jul 14 02:42:43.584701 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.584380    2584 scope.go:117] "RemoveContainer" containerID="8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7"
Jul 14 02:42:43.593409 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.593392    2584 scope.go:117] "RemoveContainer" containerID="8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7"
Jul 14 02:42:43.593667 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:42:43.593649    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7\": container with ID starting with 8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7 not found: ID does not exist" containerID="8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7"
Jul 14 02:42:43.593732 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.593680    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7"} err="failed to get container status \"8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7\": rpc error: code = NotFound desc = could not find container \"8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7\": container with ID starting with 8aacbed220250ddd23c71475d2194efc43d84411af346c572f39d3bc9f0491a7 not found: ID does not exist"
Jul 14 02:42:43.603720 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.603699    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"]
Jul 14 02:42:43.609623 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:43.609599    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f83e72-564dd855fd-xfl7x"]
Jul 14 02:42:44.891464 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:42:44.891430    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" path="/var/lib/kubelet/pods/eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737/volumes"
Jul 14 02:43:02.086976 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.086942    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"]
Jul 14 02:43:02.087768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.087416    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" containerName="maas-api"
Jul 14 02:43:02.087768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.087433    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" containerName="maas-api"
Jul 14 02:43:02.087768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.087444    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" containerName="maas-api"
Jul 14 02:43:02.087768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.087449    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" containerName="maas-api"
Jul 14 02:43:02.087768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.087518    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9199eb-ad2e-4f3d-92ed-45fe3b0e8737" containerName="maas-api"
Jul 14 02:43:02.087768 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.087530    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="06cc7c65-8fe8-4f19-88ec-2e5b51c6d1dd" containerName="maas-api"
Jul 14 02:43:02.090331 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.090314    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.093052 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.093033    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a148c7-serving-cert\""
Jul 14 02:43:02.100696 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.100668    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"]
Jul 14 02:43:02.175699 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.175662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9273dd7-a68c-4393-b714-1b794a0aa44d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.175882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.175818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkh9g\" (UniqueName: \"kubernetes.io/projected/c9273dd7-a68c-4393-b714-1b794a0aa44d-kube-api-access-zkh9g\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.175882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.175860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9273dd7-a68c-4393-b714-1b794a0aa44d-maas-api-tls\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.276512 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.276466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9273dd7-a68c-4393-b714-1b794a0aa44d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.276683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.276555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkh9g\" (UniqueName: \"kubernetes.io/projected/c9273dd7-a68c-4393-b714-1b794a0aa44d-kube-api-access-zkh9g\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.276683 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.276576    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9273dd7-a68c-4393-b714-1b794a0aa44d-maas-api-tls\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.277365 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.277328    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9273dd7-a68c-4393-b714-1b794a0aa44d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.279834 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.279804    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9273dd7-a68c-4393-b714-1b794a0aa44d-maas-api-tls\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.284299 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.284274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkh9g\" (UniqueName: \"kubernetes.io/projected/c9273dd7-a68c-4393-b714-1b794a0aa44d-kube-api-access-zkh9g\") pod \"maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.402285 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.402257    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:02.538067 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.538043    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"]
Jul 14 02:43:02.540339 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:43:02.540302    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9273dd7_a68c_4393_b714_1b794a0aa44d.slice/crio-0fbb939fd4ab2c14daf6297826e3d5d83ce1ee59c0a417219fe4960082044aa6 WatchSource:0}: Error finding container 0fbb939fd4ab2c14daf6297826e3d5d83ce1ee59c0a417219fe4960082044aa6: Status 404 returned error can't find the container with id 0fbb939fd4ab2c14daf6297826e3d5d83ce1ee59c0a417219fe4960082044aa6
Jul 14 02:43:02.671055 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:02.670964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh" event={"ID":"c9273dd7-a68c-4393-b714-1b794a0aa44d","Type":"ContainerStarted","Data":"0fbb939fd4ab2c14daf6297826e3d5d83ce1ee59c0a417219fe4960082044aa6"}
Jul 14 02:43:03.677428 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:03.677395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh" event={"ID":"c9273dd7-a68c-4393-b714-1b794a0aa44d","Type":"ContainerStarted","Data":"35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f"}
Jul 14 02:43:03.677826 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:03.677511    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:03.694171 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:03.694111    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh" podStartSLOduration=1.196349342 podStartE2EDuration="1.69409858s" totalImagesPullingTime="497.749238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:43:03.692331616 +0000 UTC m=+2477.365206429" watchObservedRunningTime="2026-07-14 02:43:03.69409858 +0000 UTC m=+2477.366973397"
Jul 14 02:43:09.687606 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:09.687576    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:43:10.239211 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.239179    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"]
Jul 14 02:43:10.243172 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.243149    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.245641 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.245617    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c47335-serving-cert\""
Jul 14 02:43:10.250774 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.250746    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"]
Jul 14 02:43:10.333850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.333810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.334029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.333877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mm8\" (UniqueName: \"kubernetes.io/projected/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-kube-api-access-95mm8\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.334029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.333927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.435105 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.435057    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mm8\" (UniqueName: \"kubernetes.io/projected/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-kube-api-access-95mm8\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.435324 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.435170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.435324 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.435262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.435451 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:43:10.435368    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-serving-cert: secret "maas-api-e2e-shared-b-c47335-serving-cert" not found
Jul 14 02:43:10.435451 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:43:10.435438    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls podName:3c7cdbb5-4f23-44d6-a8ce-7569bee7f696 nodeName:}" failed. No retries permitted until 2026-07-14 02:43:10.935417384 +0000 UTC m=+2484.608292167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls") pod "maas-api-e2e-shared-b-c47335-5455848969-hh5bq" (UID: "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696") : secret "maas-api-e2e-shared-b-c47335-serving-cert" not found
Jul 14 02:43:10.435897 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.435872    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.445687 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.445662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mm8\" (UniqueName: \"kubernetes.io/projected/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-kube-api-access-95mm8\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.938641 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.938608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:10.941013 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:10.940993    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls\") pod \"maas-api-e2e-shared-b-c47335-5455848969-hh5bq\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:11.156029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:11.155989    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:11.289189 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:11.289071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"]
Jul 14 02:43:11.291840 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:43:11.291813    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7cdbb5_4f23_44d6_a8ce_7569bee7f696.slice/crio-c11baeaadc95234079b083b15bd4c1736ccad49e355201b791b62b3eaf01bcec WatchSource:0}: Error finding container c11baeaadc95234079b083b15bd4c1736ccad49e355201b791b62b3eaf01bcec: Status 404 returned error can't find the container with id c11baeaadc95234079b083b15bd4c1736ccad49e355201b791b62b3eaf01bcec
Jul 14 02:43:11.719373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:11.718494    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq" event={"ID":"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696","Type":"ContainerStarted","Data":"6a1ec08472aede5321bcab98387474529cfaeef23e52fd3d6073e8805eeac529"}
Jul 14 02:43:11.719373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:11.718586    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq" event={"ID":"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696","Type":"ContainerStarted","Data":"c11baeaadc95234079b083b15bd4c1736ccad49e355201b791b62b3eaf01bcec"}
Jul 14 02:43:11.719373 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:11.719291    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:43:11.736004 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:11.735960    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq" podStartSLOduration=1.381213494 podStartE2EDuration="1.73594729s" totalImagesPullingTime="354.733796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:43:11.73305932 +0000 UTC m=+2485.405934119" watchObservedRunningTime="2026-07-14 02:43:11.73594729 +0000 UTC m=+2485.408822089"
Jul 14 02:43:18.734056 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:43:18.734028    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:44:07.895687 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.895652    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"]
Jul 14 02:44:07.899028 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.899010    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:07.901588 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.901567    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c47335\"/\"kube-root-ca.crt\""
Jul 14 02:44:07.901728 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.901625    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c47335\"/\"openshift-service-ca.crt\""
Jul 14 02:44:07.902791 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.902770    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c47335\"/\"default-dockercfg-ntnb9\""
Jul 14 02:44:07.903036 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.903014    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c47335\"/\"auth-test-model-c47335-kserve-self-signed-certs\""
Jul 14 02:44:07.908408 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.908388    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"]
Jul 14 02:44:07.919514 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.919493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19318027-61f5-411f-8866-04a64cd72ed4-tls-certs\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:07.919624 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.919528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-home\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:07.919624 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.919555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-tmp-dir\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:07.919752 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.919637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-model-cache\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:07.919752 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.919673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-dshm\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:07.919752 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:07.919736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnx4\" (UniqueName: \"kubernetes.io/projected/19318027-61f5-411f-8866-04a64cd72ed4-kube-api-access-dbnx4\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.020515 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-home\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.020695 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-tmp-dir\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.020695 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-model-cache\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.020695 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-dshm\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.020695 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnx4\" (UniqueName: \"kubernetes.io/projected/19318027-61f5-411f-8866-04a64cd72ed4-kube-api-access-dbnx4\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.020695 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19318027-61f5-411f-8866-04a64cd72ed4-tls-certs\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.021097 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020896    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-tmp-dir\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.021097 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.020957    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-home\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.021097 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.021051    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-model-cache\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.022867 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.022837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-dshm\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.023050 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.023035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19318027-61f5-411f-8866-04a64cd72ed4-tls-certs\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.028139 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.028114    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnx4\" (UniqueName: \"kubernetes.io/projected/19318027-61f5-411f-8866-04a64cd72ed4-kube-api-access-dbnx4\") pod \"auth-test-model-c47335-kserve-584d77dbf9-cz5gk\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") " pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.209969 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.209883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:08.335693 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.335639    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"]
Jul 14 02:44:08.338437 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:44:08.338403    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19318027_61f5_411f_8866_04a64cd72ed4.slice/crio-d69b32d28061d60bec6c10d5ab996422d44dcdb7a926abbfdab2a7009aa09955 WatchSource:0}: Error finding container d69b32d28061d60bec6c10d5ab996422d44dcdb7a926abbfdab2a7009aa09955: Status 404 returned error can't find the container with id d69b32d28061d60bec6c10d5ab996422d44dcdb7a926abbfdab2a7009aa09955
Jul 14 02:44:08.968311 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.968277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk" event={"ID":"19318027-61f5-411f-8866-04a64cd72ed4","Type":"ContainerStarted","Data":"4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7"}
Jul 14 02:44:08.968687 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.968319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk" event={"ID":"19318027-61f5-411f-8866-04a64cd72ed4","Type":"ContainerStarted","Data":"d69b32d28061d60bec6c10d5ab996422d44dcdb7a926abbfdab2a7009aa09955"}
Jul 14 02:44:08.984991 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:08.984938    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk" podStartSLOduration=1.984921739 podStartE2EDuration="1.984921739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:44:08.982506957 +0000 UTC m=+2542.655381756" watchObservedRunningTime="2026-07-14 02:44:08.984921739 +0000 UTC m=+2542.657796539"
Jul 14 02:44:18.210858 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:18.210826    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:18.211336 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:18.211072    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:18.223581 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:18.223556    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:44:19.022055 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:44:19.022026    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:46:10.492290 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:10.492258    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"]
Jul 14 02:46:25.513029 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.512963    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk" podUID="19318027-61f5-411f-8866-04a64cd72ed4" containerName="main" containerID="cri-o://4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7" gracePeriod=45
Jul 14 02:46:25.767658 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.767601    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:46:25.903201 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903170    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbnx4\" (UniqueName: \"kubernetes.io/projected/19318027-61f5-411f-8866-04a64cd72ed4-kube-api-access-dbnx4\") pod \"19318027-61f5-411f-8866-04a64cd72ed4\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") "
Jul 14 02:46:25.903381 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903218    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-model-cache\") pod \"19318027-61f5-411f-8866-04a64cd72ed4\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") "
Jul 14 02:46:25.903381 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903260    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19318027-61f5-411f-8866-04a64cd72ed4-tls-certs\") pod \"19318027-61f5-411f-8866-04a64cd72ed4\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") "
Jul 14 02:46:25.903381 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903305    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-tmp-dir\") pod \"19318027-61f5-411f-8866-04a64cd72ed4\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") "
Jul 14 02:46:25.903381 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903325    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-home\") pod \"19318027-61f5-411f-8866-04a64cd72ed4\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") "
Jul 14 02:46:25.903381 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903346    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-dshm\") pod \"19318027-61f5-411f-8866-04a64cd72ed4\" (UID: \"19318027-61f5-411f-8866-04a64cd72ed4\") "
Jul 14 02:46:25.903643 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903552    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19318027-61f5-411f-8866-04a64cd72ed4" (UID: "19318027-61f5-411f-8866-04a64cd72ed4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:46:25.903643 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903565    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-model-cache" (OuterVolumeSpecName: "model-cache") pod "19318027-61f5-411f-8866-04a64cd72ed4" (UID: "19318027-61f5-411f-8866-04a64cd72ed4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:46:25.903643 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.903581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-home" (OuterVolumeSpecName: "home") pod "19318027-61f5-411f-8866-04a64cd72ed4" (UID: "19318027-61f5-411f-8866-04a64cd72ed4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:46:25.905443 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.905417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19318027-61f5-411f-8866-04a64cd72ed4-kube-api-access-dbnx4" (OuterVolumeSpecName: "kube-api-access-dbnx4") pod "19318027-61f5-411f-8866-04a64cd72ed4" (UID: "19318027-61f5-411f-8866-04a64cd72ed4"). InnerVolumeSpecName "kube-api-access-dbnx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:46:25.905443 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.905433    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-dshm" (OuterVolumeSpecName: "dshm") pod "19318027-61f5-411f-8866-04a64cd72ed4" (UID: "19318027-61f5-411f-8866-04a64cd72ed4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:46:25.905607 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:25.905436    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19318027-61f5-411f-8866-04a64cd72ed4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19318027-61f5-411f-8866-04a64cd72ed4" (UID: "19318027-61f5-411f-8866-04a64cd72ed4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:46:26.004647 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.004619    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnx4\" (UniqueName: \"kubernetes.io/projected/19318027-61f5-411f-8866-04a64cd72ed4-kube-api-access-dbnx4\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:46:26.004647 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.004644    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:46:26.004647 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.004654    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19318027-61f5-411f-8866-04a64cd72ed4-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:46:26.004850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.004682    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:46:26.004850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.004690    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:46:26.004850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.004698    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19318027-61f5-411f-8866-04a64cd72ed4-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:46:26.570738 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.570702    2584 generic.go:356] "Generic (PLEG): container finished" podID="19318027-61f5-411f-8866-04a64cd72ed4" containerID="4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7" exitCode=0
Jul 14 02:46:26.570738 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.570741    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk" event={"ID":"19318027-61f5-411f-8866-04a64cd72ed4","Type":"ContainerDied","Data":"4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7"}
Jul 14 02:46:26.571314 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.570745    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"
Jul 14 02:46:26.571314 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.570773    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk" event={"ID":"19318027-61f5-411f-8866-04a64cd72ed4","Type":"ContainerDied","Data":"d69b32d28061d60bec6c10d5ab996422d44dcdb7a926abbfdab2a7009aa09955"}
Jul 14 02:46:26.571314 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.570793    2584 scope.go:117] "RemoveContainer" containerID="4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7"
Jul 14 02:46:26.580178 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.580153    2584 scope.go:117] "RemoveContainer" containerID="4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7"
Jul 14 02:46:26.580432 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:46:26.580416    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7\": container with ID starting with 4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7 not found: ID does not exist" containerID="4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7"
Jul 14 02:46:26.580481 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.580441    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7"} err="failed to get container status \"4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7\": rpc error: code = NotFound desc = could not find container \"4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7\": container with ID starting with 4730421f164e6e147917f6dd871bbb9b706f30be04d0b9165f15ba8e98b871a7 not found: ID does not exist"
Jul 14 02:46:26.591744 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.591722    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"]
Jul 14 02:46:26.593271 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.593254    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c47335/auth-test-model-c47335-kserve-584d77dbf9-cz5gk"]
Jul 14 02:46:26.891460 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:26.891434    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19318027-61f5-411f-8866-04a64cd72ed4" path="/var/lib/kubelet/pods/19318027-61f5-411f-8866-04a64cd72ed4/volumes"
Jul 14 02:46:53.892035 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.891991    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"]
Jul 14 02:46:53.892639 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.892391    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19318027-61f5-411f-8866-04a64cd72ed4" containerName="main"
Jul 14 02:46:53.892639 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.892403    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="19318027-61f5-411f-8866-04a64cd72ed4" containerName="main"
Jul 14 02:46:53.892639 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.892477    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="19318027-61f5-411f-8866-04a64cd72ed4" containerName="main"
Jul 14 02:46:53.896478 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.896458    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:53.900336 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.900312    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"kube-root-ca.crt\""
Jul 14 02:46:53.900445 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.900349    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"openshift-service-ca.crt\""
Jul 14 02:46:53.900445 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.900403    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"sub-test-model-a148c7-kserve-self-signed-certs\""
Jul 14 02:46:53.900531 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.900523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"default-dockercfg-888g6\""
Jul 14 02:46:53.904314 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:53.904288    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"]
Jul 14 02:46:54.043411 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.043377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tls-certs\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.043591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.043433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tmp-dir\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.043591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.043454    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p6rx\" (UniqueName: \"kubernetes.io/projected/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-kube-api-access-5p6rx\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.043591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.043521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-dshm\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.043591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.043566    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-home\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.043591 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.043586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-model-cache\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.144925 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.144825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tls-certs\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.144925 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.144919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tmp-dir\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145190 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.144952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p6rx\" (UniqueName: \"kubernetes.io/projected/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-kube-api-access-5p6rx\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145190 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.144985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-dshm\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145190 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.145022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-home\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145190 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.145047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-model-cache\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145424 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.145372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tmp-dir\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145482 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.145433    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-model-cache\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.145520 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.145483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-home\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.147209 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.147187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-dshm\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.147436 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.147417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tls-certs\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.151971 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.151952    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p6rx\" (UniqueName: \"kubernetes.io/projected/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-kube-api-access-5p6rx\") pod \"sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") " pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.206954 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.206926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:46:54.333500 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.333478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"]
Jul 14 02:46:54.335660 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:46:54.335624    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf01a3b17_25d7_45e6_94d3_98c4cb7d4856.slice/crio-eba7c441c6751ff3e0ab1c767bdb0068869b94485bb421e6a42fcf3c66b48df6 WatchSource:0}: Error finding container eba7c441c6751ff3e0ab1c767bdb0068869b94485bb421e6a42fcf3c66b48df6: Status 404 returned error can't find the container with id eba7c441c6751ff3e0ab1c767bdb0068869b94485bb421e6a42fcf3c66b48df6
Jul 14 02:46:54.701060 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.701027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p" event={"ID":"f01a3b17-25d7-45e6-94d3-98c4cb7d4856","Type":"ContainerStarted","Data":"69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b"}
Jul 14 02:46:54.701248 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.701068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p" event={"ID":"f01a3b17-25d7-45e6-94d3-98c4cb7d4856","Type":"ContainerStarted","Data":"eba7c441c6751ff3e0ab1c767bdb0068869b94485bb421e6a42fcf3c66b48df6"}
Jul 14 02:46:54.717533 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:46:54.717484    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p" podStartSLOduration=1.7174679990000001 podStartE2EDuration="1.717467999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:46:54.716123866 +0000 UTC m=+2708.388998667" watchObservedRunningTime="2026-07-14 02:46:54.717467999 +0000 UTC m=+2708.390342798"
Jul 14 02:47:04.207720 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:47:04.207691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:47:04.208248 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:47:04.207739    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:47:04.220306 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:47:04.220283    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:47:04.757612 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:47:04.757579    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:47:51.937271 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:47:51.937239    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"]
Jul 14 02:48:06.958215 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:06.958163    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p" podUID="f01a3b17-25d7-45e6-94d3-98c4cb7d4856" containerName="main" containerID="cri-o://69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b" gracePeriod=45
Jul 14 02:48:07.204498 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.204475    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:48:07.390347 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390326    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-home\") pod \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") "
Jul 14 02:48:07.390489 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390406    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tls-certs\") pod \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") "
Jul 14 02:48:07.390489 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390423    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-model-cache\") pod \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") "
Jul 14 02:48:07.390489 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390450    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p6rx\" (UniqueName: \"kubernetes.io/projected/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-kube-api-access-5p6rx\") pod \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") "
Jul 14 02:48:07.390624 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390495    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tmp-dir\") pod \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") "
Jul 14 02:48:07.390624 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390522    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-dshm\") pod \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\" (UID: \"f01a3b17-25d7-45e6-94d3-98c4cb7d4856\") "
Jul 14 02:48:07.390624 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390564    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-home" (OuterVolumeSpecName: "home") pod "f01a3b17-25d7-45e6-94d3-98c4cb7d4856" (UID: "f01a3b17-25d7-45e6-94d3-98c4cb7d4856"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:48:07.390786 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390690    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-model-cache" (OuterVolumeSpecName: "model-cache") pod "f01a3b17-25d7-45e6-94d3-98c4cb7d4856" (UID: "f01a3b17-25d7-45e6-94d3-98c4cb7d4856"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:48:07.390840 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390803    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:48:07.390840 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390822    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:48:07.390840 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.390795    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f01a3b17-25d7-45e6-94d3-98c4cb7d4856" (UID: "f01a3b17-25d7-45e6-94d3-98c4cb7d4856"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:48:07.392533 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.392502    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-kube-api-access-5p6rx" (OuterVolumeSpecName: "kube-api-access-5p6rx") pod "f01a3b17-25d7-45e6-94d3-98c4cb7d4856" (UID: "f01a3b17-25d7-45e6-94d3-98c4cb7d4856"). InnerVolumeSpecName "kube-api-access-5p6rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:48:07.392533 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.392520    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f01a3b17-25d7-45e6-94d3-98c4cb7d4856" (UID: "f01a3b17-25d7-45e6-94d3-98c4cb7d4856"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:48:07.392676 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.392518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-dshm" (OuterVolumeSpecName: "dshm") pod "f01a3b17-25d7-45e6-94d3-98c4cb7d4856" (UID: "f01a3b17-25d7-45e6-94d3-98c4cb7d4856"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:48:07.491709 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.491673    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:48:07.491709 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.491700    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p6rx\" (UniqueName: \"kubernetes.io/projected/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-kube-api-access-5p6rx\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:48:07.491709 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.491710    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:48:07.491709 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:07.491719    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01a3b17-25d7-45e6-94d3-98c4cb7d4856-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:48:08.032418 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.032382    2584 generic.go:356] "Generic (PLEG): container finished" podID="f01a3b17-25d7-45e6-94d3-98c4cb7d4856" containerID="69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b" exitCode=0
Jul 14 02:48:08.032823 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.032434    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"
Jul 14 02:48:08.032823 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.032459    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p" event={"ID":"f01a3b17-25d7-45e6-94d3-98c4cb7d4856","Type":"ContainerDied","Data":"69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b"}
Jul 14 02:48:08.032823 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.032506    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p" event={"ID":"f01a3b17-25d7-45e6-94d3-98c4cb7d4856","Type":"ContainerDied","Data":"eba7c441c6751ff3e0ab1c767bdb0068869b94485bb421e6a42fcf3c66b48df6"}
Jul 14 02:48:08.032823 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.032525    2584 scope.go:117] "RemoveContainer" containerID="69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b"
Jul 14 02:48:08.043494 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.043469    2584 scope.go:117] "RemoveContainer" containerID="69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b"
Jul 14 02:48:08.043871 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:48:08.043846    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b\": container with ID starting with 69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b not found: ID does not exist" containerID="69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b"
Jul 14 02:48:08.043934 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.043881    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b"} err="failed to get container status \"69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b\": rpc error: code = NotFound desc = could not find container \"69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b\": container with ID starting with 69648ceb1c8118ce463f85629263f374ff8db89cc7a5035c875724d56ba5100b not found: ID does not exist"
Jul 14 02:48:08.056145 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.056109    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"]
Jul 14 02:48:08.057819 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.057800    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/sub-test-model-a148c7-kserve-54cdf69cdc-t9t8p"]
Jul 14 02:48:08.891794 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:08.891719    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f01a3b17-25d7-45e6-94d3-98c4cb7d4856" path="/var/lib/kubelet/pods/f01a3b17-25d7-45e6-94d3-98c4cb7d4856/volumes"
Jul 14 02:48:21.097223 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.097184    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"]
Jul 14 02:48:21.097605 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.097579    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01a3b17-25d7-45e6-94d3-98c4cb7d4856" containerName="main"
Jul 14 02:48:21.097605 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.097593    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01a3b17-25d7-45e6-94d3-98c4cb7d4856" containerName="main"
Jul 14 02:48:21.097681 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.097645    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01a3b17-25d7-45e6-94d3-98c4cb7d4856" containerName="main"
Jul 14 02:48:21.106819 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.106793    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.107542 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.107500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"]
Jul 14 02:48:21.109571 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.109543    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"kube-root-ca.crt\""
Jul 14 02:48:21.110632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.110601    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"openshift-service-ca.crt\""
Jul 14 02:48:21.110632 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.110622    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"rate-test-model-a148c7-kserve-self-signed-certs\""
Jul 14 02:48:21.110805 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.110628    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a148c7\"/\"default-dockercfg-888g6\""
Jul 14 02:48:21.206374 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.206335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-dshm\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.206549 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.206385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2bj\" (UniqueName: \"kubernetes.io/projected/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-kube-api-access-xd2bj\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.206549 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.206488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-home\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.206549 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.206516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tmp-dir\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.206549 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.206532    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tls-certs\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.206764 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.206558    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-model-cache\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.307534 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.307497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-dshm\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.307534 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.307541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2bj\" (UniqueName: \"kubernetes.io/projected/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-kube-api-access-xd2bj\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.307740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.307583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-home\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.307740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.307603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tmp-dir\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.307740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.307620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tls-certs\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.307740 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.307644    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-model-cache\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.308049 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.308022    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-home\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.308152 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.308030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tmp-dir\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.308152 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.308105    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-model-cache\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.309850 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.309828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-dshm\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.310036 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.310020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tls-certs\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.314804 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.314781    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2bj\" (UniqueName: \"kubernetes.io/projected/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-kube-api-access-xd2bj\") pod \"rate-test-model-a148c7-kserve-7c56c4966c-qvfvg\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") " pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.417219 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.417186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:21.543805 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:21.543780    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"]
Jul 14 02:48:21.546177 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:48:21.546151    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ffe71e_7a4e_4d79_9549_4c6a15bf0e81.slice/crio-a6ac66ee88f76ffc35461a3c756a54edcbf33e6a31d7bdfc1a8bf9bb3a26bd61 WatchSource:0}: Error finding container a6ac66ee88f76ffc35461a3c756a54edcbf33e6a31d7bdfc1a8bf9bb3a26bd61: Status 404 returned error can't find the container with id a6ac66ee88f76ffc35461a3c756a54edcbf33e6a31d7bdfc1a8bf9bb3a26bd61
Jul 14 02:48:22.111846 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:22.111807    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg" event={"ID":"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81","Type":"ContainerStarted","Data":"80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505"}
Jul 14 02:48:22.111846 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:22.111848    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg" event={"ID":"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81","Type":"ContainerStarted","Data":"a6ac66ee88f76ffc35461a3c756a54edcbf33e6a31d7bdfc1a8bf9bb3a26bd61"}
Jul 14 02:48:22.126671 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:22.126623    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg" podStartSLOduration=1.126607878 podStartE2EDuration="1.126607878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 02:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:48:22.126462096 +0000 UTC m=+2795.799336887" watchObservedRunningTime="2026-07-14 02:48:22.126607878 +0000 UTC m=+2795.799482655"
Jul 14 02:48:31.417485 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:31.417450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:31.417978 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:31.417498    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:31.430055 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:31.430032    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:48:32.169969 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:48:32.169941    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:49:22.381490 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:22.381448    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"]
Jul 14 02:49:37.403232 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.403191    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg" podUID="37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" containerName="main" containerID="cri-o://80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505" gracePeriod=45
Jul 14 02:49:37.663648 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.663588    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:49:37.782502 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782475    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-home\") pod \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") "
Jul 14 02:49:37.782663 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782515    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd2bj\" (UniqueName: \"kubernetes.io/projected/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-kube-api-access-xd2bj\") pod \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") "
Jul 14 02:49:37.782663 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782573    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-dshm\") pod \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") "
Jul 14 02:49:37.782663 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782604    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tls-certs\") pod \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") "
Jul 14 02:49:37.782789 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782747    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-model-cache\") pod \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") "
Jul 14 02:49:37.782789 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782776    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-home" (OuterVolumeSpecName: "home") pod "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" (UID: "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:49:37.782867 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.782804    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tmp-dir\") pod \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\" (UID: \"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81\") "
Jul 14 02:49:37.783061 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.783008    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-model-cache" (OuterVolumeSpecName: "model-cache") pod "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" (UID: "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:49:37.783234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.783065    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" (UID: "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:49:37.783234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.783177    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-home\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:37.783234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.783191    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-model-cache\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:37.783234 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.783204    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tmp-dir\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:37.784528 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.784508    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-dshm" (OuterVolumeSpecName: "dshm") pod "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" (UID: "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 02:49:37.785046 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.785027    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" (UID: "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:49:37.785103 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.785057    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-kube-api-access-xd2bj" (OuterVolumeSpecName: "kube-api-access-xd2bj") pod "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" (UID: "37ffe71e-7a4e-4d79-9549-4c6a15bf0e81"). InnerVolumeSpecName "kube-api-access-xd2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:49:37.883579 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.883547    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2bj\" (UniqueName: \"kubernetes.io/projected/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-kube-api-access-xd2bj\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:37.883579 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.883572    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-dshm\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:37.883579 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:37.883583    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81-tls-certs\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:38.456472 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.456387    2584 generic.go:356] "Generic (PLEG): container finished" podID="37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" containerID="80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505" exitCode=0
Jul 14 02:49:38.456472 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.456434    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"
Jul 14 02:49:38.456985 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.456472    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg" event={"ID":"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81","Type":"ContainerDied","Data":"80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505"}
Jul 14 02:49:38.456985 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.456517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg" event={"ID":"37ffe71e-7a4e-4d79-9549-4c6a15bf0e81","Type":"ContainerDied","Data":"a6ac66ee88f76ffc35461a3c756a54edcbf33e6a31d7bdfc1a8bf9bb3a26bd61"}
Jul 14 02:49:38.456985 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.456538    2584 scope.go:117] "RemoveContainer" containerID="80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505"
Jul 14 02:49:38.466229 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.466212    2584 scope.go:117] "RemoveContainer" containerID="80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505"
Jul 14 02:49:38.466476 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:49:38.466458    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505\": container with ID starting with 80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505 not found: ID does not exist" containerID="80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505"
Jul 14 02:49:38.466526 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.466486    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505"} err="failed to get container status \"80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505\": rpc error: code = NotFound desc = could not find container \"80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505\": container with ID starting with 80c2f3ebb5d941e97c49c44ffa16dfc24f425259237279e67d612efbd7ea1505 not found: ID does not exist"
Jul 14 02:49:38.477991 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.477966    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"]
Jul 14 02:49:38.483039 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.483018    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a148c7/rate-test-model-a148c7-kserve-7c56c4966c-qvfvg"]
Jul 14 02:49:38.896813 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:38.896783    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" path="/var/lib/kubelet/pods/37ffe71e-7a4e-4d79-9549-4c6a15bf0e81/volumes"
Jul 14 02:49:53.161515 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.160754    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"]
Jul 14 02:49:53.161515 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.161028    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh" podUID="c9273dd7-a68c-4393-b714-1b794a0aa44d" containerName="maas-api" containerID="cri-o://35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f" gracePeriod=30
Jul 14 02:49:53.415712 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.415657    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:49:53.521068 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.521034    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkh9g\" (UniqueName: \"kubernetes.io/projected/c9273dd7-a68c-4393-b714-1b794a0aa44d-kube-api-access-zkh9g\") pod \"c9273dd7-a68c-4393-b714-1b794a0aa44d\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") "
Jul 14 02:49:53.521260 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.521070    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9273dd7-a68c-4393-b714-1b794a0aa44d-maas-api-tls\") pod \"c9273dd7-a68c-4393-b714-1b794a0aa44d\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") "
Jul 14 02:49:53.521260 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.521149    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9273dd7-a68c-4393-b714-1b794a0aa44d-openshift-service-ca\") pod \"c9273dd7-a68c-4393-b714-1b794a0aa44d\" (UID: \"c9273dd7-a68c-4393-b714-1b794a0aa44d\") "
Jul 14 02:49:53.521524 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.521502    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9273dd7-a68c-4393-b714-1b794a0aa44d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9273dd7-a68c-4393-b714-1b794a0aa44d" (UID: "c9273dd7-a68c-4393-b714-1b794a0aa44d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:49:53.523233 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.523188    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9273dd7-a68c-4393-b714-1b794a0aa44d-kube-api-access-zkh9g" (OuterVolumeSpecName: "kube-api-access-zkh9g") pod "c9273dd7-a68c-4393-b714-1b794a0aa44d" (UID: "c9273dd7-a68c-4393-b714-1b794a0aa44d"). InnerVolumeSpecName "kube-api-access-zkh9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:49:53.523330 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.523241    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9273dd7-a68c-4393-b714-1b794a0aa44d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9273dd7-a68c-4393-b714-1b794a0aa44d" (UID: "c9273dd7-a68c-4393-b714-1b794a0aa44d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:49:53.527777 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.527750    2584 generic.go:356] "Generic (PLEG): container finished" podID="c9273dd7-a68c-4393-b714-1b794a0aa44d" containerID="35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f" exitCode=0
Jul 14 02:49:53.527885 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.527796    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"
Jul 14 02:49:53.527885 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.527838    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh" event={"ID":"c9273dd7-a68c-4393-b714-1b794a0aa44d","Type":"ContainerDied","Data":"35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f"}
Jul 14 02:49:53.527885 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.527879    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh" event={"ID":"c9273dd7-a68c-4393-b714-1b794a0aa44d","Type":"ContainerDied","Data":"0fbb939fd4ab2c14daf6297826e3d5d83ce1ee59c0a417219fe4960082044aa6"}
Jul 14 02:49:53.528047 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.527899    2584 scope.go:117] "RemoveContainer" containerID="35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f"
Jul 14 02:49:53.539560 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.539542    2584 scope.go:117] "RemoveContainer" containerID="35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f"
Jul 14 02:49:53.539795 ip-10-0-128-94 kubenswrapper[2584]: E0714 02:49:53.539777    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f\": container with ID starting with 35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f not found: ID does not exist" containerID="35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f"
Jul 14 02:49:53.539862 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.539807    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f"} err="failed to get container status \"35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f\": rpc error: code = NotFound desc = could not find container \"35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f\": container with ID starting with 35e78b52ad068557d991e932163f7f325690c24fefc3eb12ac1786945916450f not found: ID does not exist"
Jul 14 02:49:53.549832 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.549810    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"]
Jul 14 02:49:53.556248 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.556228    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a148c7-bf655fbd7-jc5xh"]
Jul 14 02:49:53.621951 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.621925    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkh9g\" (UniqueName: \"kubernetes.io/projected/c9273dd7-a68c-4393-b714-1b794a0aa44d-kube-api-access-zkh9g\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:53.621951 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.621948    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9273dd7-a68c-4393-b714-1b794a0aa44d-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:53.622076 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:53.621959    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9273dd7-a68c-4393-b714-1b794a0aa44d-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:49:54.891567 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:49:54.891532    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9273dd7-a68c-4393-b714-1b794a0aa44d" path="/var/lib/kubelet/pods/c9273dd7-a68c-4393-b714-1b794a0aa44d/volumes"
Jul 14 02:50:19.224060 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.224022    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"]
Jul 14 02:50:19.226615 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.224337    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq" podUID="3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" containerName="maas-api" containerID="cri-o://6a1ec08472aede5321bcab98387474529cfaeef23e52fd3d6073e8805eeac529" gracePeriod=30
Jul 14 02:50:19.643529 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.643497    2584 generic.go:356] "Generic (PLEG): container finished" podID="3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" containerID="6a1ec08472aede5321bcab98387474529cfaeef23e52fd3d6073e8805eeac529" exitCode=0
Jul 14 02:50:19.643690 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.643579    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq" event={"ID":"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696","Type":"ContainerDied","Data":"6a1ec08472aede5321bcab98387474529cfaeef23e52fd3d6073e8805eeac529"}
Jul 14 02:50:19.677078 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.677053    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:50:19.730268 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.730240    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-openshift-service-ca\") pod \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") "
Jul 14 02:50:19.730399 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.730280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95mm8\" (UniqueName: \"kubernetes.io/projected/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-kube-api-access-95mm8\") pod \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") "
Jul 14 02:50:19.730399 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.730318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls\") pod \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\" (UID: \"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696\") "
Jul 14 02:50:19.730678 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.730651    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" (UID: "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 02:50:19.732417 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.732389    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" (UID: "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 02:50:19.732417 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.732406    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-kube-api-access-95mm8" (OuterVolumeSpecName: "kube-api-access-95mm8") pod "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" (UID: "3c7cdbb5-4f23-44d6-a8ce-7569bee7f696"). InnerVolumeSpecName "kube-api-access-95mm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 02:50:19.831176 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.831051    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mm8\" (UniqueName: \"kubernetes.io/projected/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-kube-api-access-95mm8\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:50:19.831176 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.831089    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-maas-api-tls\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:50:19.831176 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:19.831102    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696-openshift-service-ca\") on node \"ip-10-0-128-94.ec2.internal\" DevicePath \"\""
Jul 14 02:50:20.648662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:20.648627    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq" event={"ID":"3c7cdbb5-4f23-44d6-a8ce-7569bee7f696","Type":"ContainerDied","Data":"c11baeaadc95234079b083b15bd4c1736ccad49e355201b791b62b3eaf01bcec"}
Jul 14 02:50:20.648662 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:20.648651    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"
Jul 14 02:50:20.649177 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:20.648689    2584 scope.go:117] "RemoveContainer" containerID="6a1ec08472aede5321bcab98387474529cfaeef23e52fd3d6073e8805eeac529"
Jul 14 02:50:20.670442 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:20.670419    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"]
Jul 14 02:50:20.674147 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:20.674110    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c47335-5455848969-hh5bq"]
Jul 14 02:50:20.891267 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:20.891233    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" path="/var/lib/kubelet/pods/3c7cdbb5-4f23-44d6-a8ce-7569bee7f696/volumes"
Jul 14 02:50:40.400932 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:40.400900    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cd5db494-ctxrl_c111897d-f0ea-4300-bc6a-be33476a0285/manager/0.log"
Jul 14 02:50:40.515633 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:40.515607    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pjbq5_8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a/server/0.log"
Jul 14 02:50:40.631572 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:40.631538    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c6bxl_3c950f70-bedc-4b2c-9944-8e345814dabe/manager/0.log"
Jul 14 02:50:40.992062 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:40.992034    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5756559977-t8sfj_44dd7d23-a748-44a0-8cbb-15a010bef950/copy-manifests/0.log"
Jul 14 02:50:41.002828 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:41.002804    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5756559977-t8sfj_44dd7d23-a748-44a0-8cbb-15a010bef950/manager/0.log"
Jul 14 02:50:41.360218 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:41.360150    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d9498bc97-zdqqm_83d05146-79a1-4c52-b3e4-d0d6fbc45616/maas-api/0.log"
Jul 14 02:50:43.131365 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:43.131342    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-d7mnn_174b9e78-8a3d-4e45-97d5-bc97feeac0a5/postgres/0.log"
Jul 14 02:50:44.512798 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:44.512765    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-g5fmq_21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e/manager/0.log"
Jul 14 02:50:44.859567 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:44.859488    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4gmpt_93a4f01a-afd0-4db5-904c-cdecc5fd29d2/registry-server/0.log"
Jul 14 02:50:45.201239 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:45.201209    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bbqp9_9b72815e-a93c-4194-bb2e-ed818d6b36c4/manager/0.log"
Jul 14 02:50:45.682045 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:45.682017    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d5grx_aa0730b6-12a1-4d73-a6ae-77e973f8c011/discovery/0.log"
Jul 14 02:50:45.910562 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:45.910533    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd74468c-czkql_dfc457de-e041-4eef-b19a-6ac4a064e66c/kube-auth-proxy/0.log"
Jul 14 02:50:46.144930 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:46.144902    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-n54pv_6e5e5845-3960-45c9-bd1f-367edcac9090/payload-pre-processing/0.log"
Jul 14 02:50:46.261714 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:46.261690    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-h7wvr_f1815e0b-97fa-40e7-af2e-39a06174c014/payload-processing/0.log"
Jul 14 02:50:46.722821 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:46.722792    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4qqlw_c1d39986-a273-49b5-9c2b-4edcac6f6d24/main/0.log"
Jul 14 02:50:47.060570 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:47.060488    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgq4zg_d3c80aa9-7909-4cc0-816c-6df79f8bc96b/main/0.log"
Jul 14 02:50:47.291731 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:47.291703    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-zxg7h_37bc92f3-f5c6-4ac6-8ef6-ff30aa6d655f/main/0.log"
Jul 14 02:50:54.568509 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:54.568477    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mfct5_adce1978-62d1-4533-a4b9-b8a4703092aa/global-pull-secret-syncer/0.log"
Jul 14 02:50:54.784914 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:54.784885    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r8sdt_8eda87a6-bdd6-4ca3-9213-43e673eec75b/konnectivity-agent/0.log"
Jul 14 02:50:54.799784 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:54.799758    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-94.ec2.internal_efbaaf541750e1cbbe62f0277567641b/haproxy/0.log"
Jul 14 02:50:59.771497 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:59.771464    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-g5fmq_21c0a6f0-bac6-4c3d-bc7a-c464f1e9d41e/manager/0.log"
Jul 14 02:50:59.851198 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:50:59.851173    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4gmpt_93a4f01a-afd0-4db5-904c-cdecc5fd29d2/registry-server/0.log"
Jul 14 02:51:00.073568 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:00.073494    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bbqp9_9b72815e-a93c-4194-bb2e-ed818d6b36c4/manager/0.log"
Jul 14 02:51:02.026878 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:02.026848    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gc2g_14de5b20-d544-4d51-8698-5cb1fd56b692/node-exporter/0.log"
Jul 14 02:51:02.044041 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:02.044016    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gc2g_14de5b20-d544-4d51-8698-5cb1fd56b692/kube-rbac-proxy/0.log"
Jul 14 02:51:02.063592 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:02.063574    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gc2g_14de5b20-d544-4d51-8698-5cb1fd56b692/init-textfile/0.log"
Jul 14 02:51:05.338032 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:05.337998    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-n2jm2_e0b4d3f1-1e96-48f1-935f-b2b9a7f5953e/download-server/0.log"
Jul 14 02:51:06.010210 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010175    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"]
Jul 14 02:51:06.010570 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010554    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" containerName="maas-api"
Jul 14 02:51:06.010570 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010571    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" containerName="maas-api"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010587    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" containerName="main"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010592    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" containerName="main"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010600    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9273dd7-a68c-4393-b714-1b794a0aa44d" containerName="maas-api"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010607    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9273dd7-a68c-4393-b714-1b794a0aa44d" containerName="maas-api"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010664    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9273dd7-a68c-4393-b714-1b794a0aa44d" containerName="maas-api"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010670    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c7cdbb5-4f23-44d6-a8ce-7569bee7f696" containerName="maas-api"
Jul 14 02:51:06.010704 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.010676    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ffe71e-7a4e-4d79-9549-4c6a15bf0e81" containerName="main"
Jul 14 02:51:06.013672 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.013651    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.016044 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.016022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-btqq2\"/\"openshift-service-ca.crt\""
Jul 14 02:51:06.016178 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.016044    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-btqq2\"/\"default-dockercfg-9lvvj\""
Jul 14 02:51:06.017445 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.017413    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-btqq2\"/\"kube-root-ca.crt\""
Jul 14 02:51:06.022651 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.022630    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"]
Jul 14 02:51:06.106329 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.106298    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-sys\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.106329 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.106331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-lib-modules\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.106535 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.106361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-proc\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.106535 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.106388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9xb\" (UniqueName: \"kubernetes.io/projected/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-kube-api-access-vs9xb\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.106535 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.106428    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-podres\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.206924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.206886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-sys\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.206924 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.206924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-lib-modules\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.206955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-proc\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.206972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9xb\" (UniqueName: \"kubernetes.io/projected/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-kube-api-access-vs9xb\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.207014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-podres\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.207030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-proc\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.207030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-sys\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207166 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.207088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-lib-modules\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.207355 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.207173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-podres\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.214805 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.214778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9xb\" (UniqueName: \"kubernetes.io/projected/46c01dc4-d89a-476e-9fb1-e20a6849ba9a-kube-api-access-vs9xb\") pod \"perf-node-gather-daemonset-tqxj8\" (UID: \"46c01dc4-d89a-476e-9fb1-e20a6849ba9a\") " pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.324027 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.323937    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.453503 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.453474    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"]
Jul 14 02:51:06.455767 ip-10-0-128-94 kubenswrapper[2584]: W0714 02:51:06.455740    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46c01dc4_d89a_476e_9fb1_e20a6849ba9a.slice/crio-fb02a9aca1b4f28736b81bd15ad4b0add3f2c70d96032e15d3c0c09bb330c788 WatchSource:0}: Error finding container fb02a9aca1b4f28736b81bd15ad4b0add3f2c70d96032e15d3c0c09bb330c788: Status 404 returned error can't find the container with id fb02a9aca1b4f28736b81bd15ad4b0add3f2c70d96032e15d3c0c09bb330c788
Jul 14 02:51:06.860004 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.859912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8" event={"ID":"46c01dc4-d89a-476e-9fb1-e20a6849ba9a","Type":"ContainerStarted","Data":"28e58a1a25c7c157821be1b0f98059b70c7c2dfc2466629f142f64158a972d68"}
Jul 14 02:51:06.860004 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.860003    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8" event={"ID":"46c01dc4-d89a-476e-9fb1-e20a6849ba9a","Type":"ContainerStarted","Data":"fb02a9aca1b4f28736b81bd15ad4b0add3f2c70d96032e15d3c0c09bb330c788"}
Jul 14 02:51:06.860249 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.860111    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:06.880329 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:06.880279    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8" podStartSLOduration=1.8802636430000002 podStartE2EDuration="1.880263643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 02:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 02:51:06.878625482 +0000 UTC m=+2960.551500283" watchObservedRunningTime="2026-07-14 02:51:06.880263643 +0000 UTC m=+2960.553138443"
Jul 14 02:51:07.012418 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:07.012391    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bs7sk_3215752b-e593-47c7-931b-1497ac32ec50/dns/0.log"
Jul 14 02:51:07.039595 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:07.039566    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bs7sk_3215752b-e593-47c7-931b-1497ac32ec50/kube-rbac-proxy/0.log"
Jul 14 02:51:07.118569 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:07.118495    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lddfv_81f850e6-9a94-4e26-9880-39802130d134/dns-node-resolver/0.log"
Jul 14 02:51:07.694563 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:07.694528    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkztz_a70b3316-2d49-4928-9428-88d957426bf7/node-ca/0.log"
Jul 14 02:51:08.691795 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:08.691720    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d5grx_aa0730b6-12a1-4d73-a6ae-77e973f8c011/discovery/0.log"
Jul 14 02:51:08.726068 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:08.726040    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd74468c-czkql_dfc457de-e041-4eef-b19a-6ac4a064e66c/kube-auth-proxy/0.log"
Jul 14 02:51:08.803589 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:08.803563    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-n54pv_6e5e5845-3960-45c9-bd1f-367edcac9090/payload-pre-processing/0.log"
Jul 14 02:51:08.829961 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:08.829939    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-h7wvr_f1815e0b-97fa-40e7-af2e-39a06174c014/payload-processing/0.log"
Jul 14 02:51:09.556493 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:09.556467    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zplpn_34e3dfd5-7708-4989-9a82-fc5c29dc0346/serve-healthcheck-canary/0.log"
Jul 14 02:51:10.053667 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:10.053634    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cd6b_53b2d150-03b0-425d-9f08-ab01b238d425/kube-rbac-proxy/0.log"
Jul 14 02:51:10.070064 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:10.070038    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cd6b_53b2d150-03b0-425d-9f08-ab01b238d425/exporter/0.log"
Jul 14 02:51:10.086955 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:10.086935    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cd6b_53b2d150-03b0-425d-9f08-ab01b238d425/extractor/0.log"
Jul 14 02:51:12.420217 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:12.420188    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cd5db494-ctxrl_c111897d-f0ea-4300-bc6a-be33476a0285/manager/0.log"
Jul 14 02:51:12.436890 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:12.436871    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pjbq5_8232901a-bbf1-4110-b0f3-a7c2fc8c7a3a/server/0.log"
Jul 14 02:51:12.490726 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:12.490695    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c6bxl_3c950f70-bedc-4b2c-9944-8e345814dabe/manager/0.log"
Jul 14 02:51:12.632394 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:12.632364    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5756559977-t8sfj_44dd7d23-a748-44a0-8cbb-15a010bef950/manager/0.log"
Jul 14 02:51:12.643902 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:12.643874    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5756559977-t8sfj_44dd7d23-a748-44a0-8cbb-15a010bef950/copy-manifests/0.log"
Jul 14 02:51:12.875327 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:12.875256    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-btqq2/perf-node-gather-daemonset-tqxj8"
Jul 14 02:51:16.123882 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:16.123849    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d5grx_aa0730b6-12a1-4d73-a6ae-77e973f8c011/discovery/0.log"
Jul 14 02:51:16.148559 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:16.148493    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd74468c-czkql_dfc457de-e041-4eef-b19a-6ac4a064e66c/kube-auth-proxy/0.log"
Jul 14 02:51:16.219002 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:16.218974    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-n54pv_6e5e5845-3960-45c9-bd1f-367edcac9090/payload-pre-processing/0.log"
Jul 14 02:51:16.236803 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:16.236775    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-h7wvr_f1815e0b-97fa-40e7-af2e-39a06174c014/payload-processing/0.log"
Jul 14 02:51:17.635832 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:17.635799    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d5grx_aa0730b6-12a1-4d73-a6ae-77e973f8c011/discovery/0.log"
Jul 14 02:51:17.664027 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:17.664006    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd74468c-czkql_dfc457de-e041-4eef-b19a-6ac4a064e66c/kube-auth-proxy/0.log"
Jul 14 02:51:17.738145 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:17.738112    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-n54pv_6e5e5845-3960-45c9-bd1f-367edcac9090/payload-pre-processing/0.log"
Jul 14 02:51:17.760154 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:17.760103    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-h7wvr_f1815e0b-97fa-40e7-af2e-39a06174c014/payload-processing/0.log"
Jul 14 02:51:20.566583 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.566555    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/kube-multus-additional-cni-plugins/0.log"
Jul 14 02:51:20.581126 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.581100    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/egress-router-binary-copy/0.log"
Jul 14 02:51:20.596275 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.596256    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/cni-plugins/0.log"
Jul 14 02:51:20.609396 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.609381    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/bond-cni-plugin/0.log"
Jul 14 02:51:20.626434 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.626418    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/routeoverride-cni/0.log"
Jul 14 02:51:20.638445 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.638428    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/whereabouts-cni-bincopy/0.log"
Jul 14 02:51:20.652785 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.652760    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzbxv_53f09255-d824-4fed-b5a4-bb594e253ada/whereabouts-cni/0.log"
Jul 14 02:51:20.708448 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.708423    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n8vwr_a067b638-6577-411c-a021-a92848fe5508/kube-multus/0.log"
Jul 14 02:51:20.752158 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.752122    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmhpt_a296f1cc-563f-4feb-96fe-c4c56ef944b9/network-metrics-daemon/0.log"
Jul 14 02:51:20.764751 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:20.764725    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmhpt_a296f1cc-563f-4feb-96fe-c4c56ef944b9/kube-rbac-proxy/0.log"
Jul 14 02:51:22.048100 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.048024    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/ovn-controller/0.log"
Jul 14 02:51:22.059154 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.059117    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/ovn-acl-logging/0.log"
Jul 14 02:51:22.074838 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.074808    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/kube-rbac-proxy-node/0.log"
Jul 14 02:51:22.089333 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.089313    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 14 02:51:22.102769 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.102745    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/northd/0.log"
Jul 14 02:51:22.117481 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.117464    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/nbdb/0.log"
Jul 14 02:51:22.131480 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.131460    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/sbdb/0.log"
Jul 14 02:51:22.176540 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:22.176514    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfrcp_1dc1b087-17d0-42b5-affa-a6eff7db68f4/ovnkube-controller/0.log"
Jul 14 02:51:23.226326 ip-10-0-128-94 kubenswrapper[2584]: I0714 02:51:23.226296    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-grzbf_e7231f7f-4e33-4571-b3ce-0d3f15a44a61/network-check-target-container/0.log"