Jul 11 03:42:35.004412 ip-10-0-140-230 systemd[1]: Starting Kubernetes Kubelet...
Jul 11 03:42:35.448116 ip-10-0-140-230 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 11 03:42:35.448116 ip-10-0-140-230 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 11 03:42:35.448116 ip-10-0-140-230 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 11 03:42:35.448116 ip-10-0-140-230 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 11 03:42:35.448116 ip-10-0-140-230 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 11 03:42:35.450520 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.450426    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 11 03:42:35.455717 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455701    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 11 03:42:35.455717 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455717    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455720    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455724    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455727    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455729    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455732    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455734    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455738    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455740    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455743    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455746    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455749    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455752    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455756    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455760    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455763    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455766    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455769    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455771    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455774    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 11 03:42:35.455782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455777    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455779    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455782    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455784    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455787    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455790    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455793    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455795    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455799    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455802    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455804    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455806    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455809    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455811    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455814    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455816    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455819    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455821    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455823    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455826    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 11 03:42:35.456301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455829    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455831    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455833    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455836    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455838    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455842    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455845    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455848    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455851    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455854    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455856    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455859    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455861    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455864    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455867    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455870    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455872    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455875    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455878    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 11 03:42:35.456782 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455881    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455883    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455886    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455888    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455892    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455895    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455898    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455900    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455902    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455905    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455908    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455911    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455913    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455916    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455918    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455920    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455923    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455925    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455928    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 11 03:42:35.457301 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455930    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455933    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455936    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455939    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455942    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455944    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455947    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455949    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455954    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455957    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455960    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455962    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455965    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455967    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455970    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455972    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455975    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455977    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455980    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.455982    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 11 03:42:35.457771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456711    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456717    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456720    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456723    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456725    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456734    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456737    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456739    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456742    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456744    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456747    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456749    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456759    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456761    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456764    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456768    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456771    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456774    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456777    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 11 03:42:35.458282 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456780    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456786    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456788    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456791    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456794    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456796    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456799    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456802    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456804    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456806    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456809    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456811    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456814    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456817    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456819    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456822    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456824    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456826    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456829    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456832    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 11 03:42:35.458754 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456834    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456837    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456839    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456842    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456844    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456847    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456849    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456852    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456854    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456856    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456859    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456861    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456864    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456867    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456869    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456872    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456886    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456889    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456892    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 11 03:42:35.459246 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456894    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456897    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456899    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456902    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456904    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456906    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456909    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456911    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456913    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456916    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456918    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456920    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456923    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456926    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456928    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456931    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456933    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456936    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456938    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456940    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 11 03:42:35.459706 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456943    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456945    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456947    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456950    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456952    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456955    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456958    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456961    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456963    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456968    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456971    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456973    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456976    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456978    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456981    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456983    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456987    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456991    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456993    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 11 03:42:35.460198 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456996    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.456998    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457429    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457435    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457437    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457440    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457443    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457446    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457449    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457451    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457453    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457456    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457460    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457464    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457467    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457470    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457473    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457475    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457478    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457481    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 11 03:42:35.460656 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457484    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457487    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457490    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457493    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457496    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457499    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457502    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457504    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457507    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457510    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457512    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457515    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457517    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457520    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457523    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457525    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457527    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457531    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457534    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457538    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 11 03:42:35.461245 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457541    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457544    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457546    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457549    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457551    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457553    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457556    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457559    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457561    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457564    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457566    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457569    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457571    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457573    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457576    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457580    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457583    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457586    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457589    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457591    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 11 03:42:35.461751 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457593    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457596    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457598    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457601    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457603    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457605    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457608    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457611    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457613    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457615    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457618    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457620    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457623    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457625    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457628    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457630    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457633    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457635    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457640    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457643    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 11 03:42:35.462258 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457645    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457648    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457650    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457653    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457655    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457658    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457660    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457663    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457665    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457667    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457670    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457672    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457676    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457678    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457681    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457683    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457686    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457688    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457691    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 11 03:42:35.462740 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457693    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.457696    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457770    2582 flags.go:64] FLAG: --address="0.0.0.0"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457778    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457784    2582 flags.go:64] FLAG: --anonymous-auth="true"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457788    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457793    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457796    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457800    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457804    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457807    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457812    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457815    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457818    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457821    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457824    2582 flags.go:64] FLAG: --cgroup-root=""
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457827    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457829    2582 flags.go:64] FLAG: --client-ca-file=""
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457832    2582 flags.go:64] FLAG: --cloud-provider="external"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457835    2582 flags.go:64] FLAG: --cluster-dns="[]"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457839    2582 flags.go:64] FLAG: --cluster-domain=""
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457842    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457845    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457848    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 11 03:42:35.463214 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457851    2582 flags.go:64] FLAG: --container-log-max-files="5"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457855    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457858    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457861    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457864    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457867    2582 flags.go:64] FLAG: --contention-profiling="false"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457870    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457873    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457876    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457879    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457883    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457886    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457889    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457892    2582 flags.go:64] FLAG: --enable-load-reader="false"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457895    2582 flags.go:64] FLAG: --enable-server="true"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457898    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457902    2582 flags.go:64] FLAG: --event-burst="100"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457905    2582 flags.go:64] FLAG: --event-qps="50"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457907    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457912    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457915    2582 flags.go:64] FLAG: --eviction-hard=""
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457918    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457922    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457924    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457927    2582 flags.go:64] FLAG: --eviction-soft=""
Jul 11 03:42:35.463778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457930    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457933    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457935    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457938    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457941    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457944    2582 flags.go:64] FLAG: --fail-swap-on="true"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457947    2582 flags.go:64] FLAG: --feature-gates=""
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457951    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457954    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457957    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457960    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457963    2582 flags.go:64] FLAG: --healthz-port="10248"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457966    2582 flags.go:64] FLAG: --help="false"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457968    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457973    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457976    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457979    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457982    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457985    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457988    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457991    2582 flags.go:64] FLAG: --image-service-endpoint=""
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457994    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.457997    2582 flags.go:64] FLAG: --kube-api-burst="100"
Jul 11 03:42:35.464395 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458000    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458003    2582 flags.go:64] FLAG: --kube-api-qps="50"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458006    2582 flags.go:64] FLAG: --kube-reserved=""
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458010    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458013    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458016    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458019    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458021    2582 flags.go:64] FLAG: --lock-file=""
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458024    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458026    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458029    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458034    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458037    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458040    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458042    2582 flags.go:64] FLAG: --logging-format="text"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458045    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458048    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458050    2582 flags.go:64] FLAG: --manifest-url=""
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458053    2582 flags.go:64] FLAG: --manifest-url-header=""
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458058    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458061    2582 flags.go:64] FLAG: --max-open-files="1000000"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458070    2582 flags.go:64] FLAG: --max-pods="110"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458073    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458077    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458079    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 11 03:42:35.464988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458082    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458085    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458088    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458091    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458114    2582 flags.go:64] FLAG: --node-status-max-images="50"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458117    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458119    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458122    2582 flags.go:64] FLAG: --pod-cidr=""
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458126    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458131    2582 flags.go:64] FLAG: --pod-manifest-path=""
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458134    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458139    2582 flags.go:64] FLAG: --pods-per-core="0"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458141    2582 flags.go:64] FLAG: --port="10250"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458144    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458148    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b7a36390d53b4ad0"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458151    2582 flags.go:64] FLAG: --qos-reserved=""
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458153    2582 flags.go:64] FLAG: --read-only-port="10255"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458157    2582 flags.go:64] FLAG: --register-node="true"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458159    2582 flags.go:64] FLAG: --register-with-taints=""
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458165    2582 flags.go:64] FLAG: --registry-burst="10"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458168    2582 flags.go:64] FLAG: --registry-qps="5"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458171    2582 flags.go:64] FLAG: --reserved-cpus=""
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458173    2582 flags.go:64] FLAG: --reserved-memory=""
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458177    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458180    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 11 03:42:35.465660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458182    2582 flags.go:64] FLAG: --rotate-certificates="false"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458185    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458188    2582 flags.go:64] FLAG: --runonce="false"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458191    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458194    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458198    2582 flags.go:64] FLAG: --seccomp-default="false"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458201    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458203    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458206    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458209    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458212    2582 flags.go:64] FLAG: --storage-driver-password="root"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458214    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458217    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458220    2582 flags.go:64] FLAG: --storage-driver-user="root"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458222    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458225    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458228    2582 flags.go:64] FLAG: --system-cgroups=""
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458231    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458237    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458240    2582 flags.go:64] FLAG: --tls-cert-file=""
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458243    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458247    2582 flags.go:64] FLAG: --tls-min-version=""
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458250    2582 flags.go:64] FLAG: --tls-private-key-file=""
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458253    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458255    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 11 03:42:35.466273 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458258    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458261    2582 flags.go:64] FLAG: --v="2"
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458265    2582 flags.go:64] FLAG: --version="false"
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458269    2582 flags.go:64] FLAG: --vmodule=""
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458272    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.458275    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458396    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458400    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458403    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458406    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458409    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458412    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458416    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458419    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458421    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458424    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458426    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458429    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458432    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458434    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458437    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 11 03:42:35.466922 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458439    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458441    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458444    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458446    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458450    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458453    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458456    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458458    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458461    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458463    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458466    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458468    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458471    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458473    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458475    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458478    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458480    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458482    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458485    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458487    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 11 03:42:35.467450 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458490    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458492    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458494    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458499    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458503    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458505    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458508    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458511    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458514    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458516    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458518    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458521    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458523    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458525    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458528    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458532    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458539    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458542    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 11 03:42:35.467926 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458546    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458550    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458553    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458556    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458558    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458561    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458564    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458567    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458570    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458572    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458574    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458577    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458579    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458582    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458584    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458587    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458589    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458593    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458595    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458598    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 11 03:42:35.468381 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458600    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458603    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458605    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458607    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458610    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458612    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458615    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458617    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458620    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458622    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458625    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458628    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458631    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458634    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458636    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458638    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458641    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458643    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458646    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458648    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 11 03:42:35.468858 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458650    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458653    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458656    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458658    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458660    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.458663    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.459518    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.466942    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.467060    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.467080    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.467090    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467181    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467186    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467189    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467192    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 11 03:42:35.469378 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467195    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467198    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467201    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467204    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467206    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467209    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467211    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467214    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467216    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467219    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467221    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467223    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467226    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467228    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467231    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467233    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467235    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467238    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467240    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467242    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 11 03:42:35.469755 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467245    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467247    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467250    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467252    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467255    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467264    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467266    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467269    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467272    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467274    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467276    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467279    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467281    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467283    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467286    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467288    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467290    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467293    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467295    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467297    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 11 03:42:35.470339 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467300    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467302    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467305    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467307    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467309    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467312    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467314    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467316    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467319    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467321    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467329    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467332    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467334    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467337    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467341    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467345    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467349    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467354    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467364    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 11 03:42:35.470860 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467368    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467371    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467373    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467376    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467378    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467381    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467383    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467386    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467388    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467391    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467393    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467396    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467398    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467401    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467403    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467405    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467408    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467410    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467412    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467415    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 11 03:42:35.471323 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467417    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467420    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467422    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467425    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467427    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467429    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467432    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467434    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467437    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467439    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467442    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467444    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467452    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467455    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467458    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 11 03:42:35.471851 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467461    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.467466    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467627    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467632    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467638    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467641    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467643    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467646    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467649    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467651    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467653    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467656    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467658    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467660    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467663    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 11 03:42:35.472233 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467665    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467668    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467670    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467672    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467675    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467677    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467679    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467681    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467684    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467686    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467689    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467691    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467694    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467696    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467699    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467707    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467710    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467712    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467714    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467717    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 11 03:42:35.472613 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467719    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467721    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467724    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467726    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467728    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467730    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467733    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467735    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467738    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467740    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467742    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467745    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467747    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467749    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467752    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467754    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467756    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467759    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467761    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467763    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 11 03:42:35.473118 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467765    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467768    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467770    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467773    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467775    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467777    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467780    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467782    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467790    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467793    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467795    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467799    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467802    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467805    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467808    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467811    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467814    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467817    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467819    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 11 03:42:35.473603 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467822    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467824    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467826    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467829    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467831    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467833    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467836    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467838    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467840    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467843    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467845    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467847    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467851    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467854    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467856    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467859    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467862    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467865    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467867    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 11 03:42:35.474056 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467870    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467872    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467875    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467884    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467887    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467889    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467891    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:35.467895    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.467900    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.468032    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.471445    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.473048    2582 server.go:1019] "Starting client certificate rotation"
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.473138    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 11 03:42:35.474572 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.473171    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 11 03:42:35.479576 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.479408    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 11 03:42:35.482700 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.482680    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 11 03:42:35.495279 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.495259    2582 log.go:25] "Validated CRI v1 runtime API"
Jul 11 03:42:35.497668 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.497656    2582 log.go:25] "Validated CRI v1 image API"
Jul 11 03:42:35.499001 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.498979    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 11 03:42:35.501729 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.501706    2582 fs.go:135] Filesystem UUIDs: map[6ce0a4f1-5792-4539-8780-e4a44933277f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 af765a16-6b80-49a3-8a8b-623d55747e94:/dev/nvme0n1p3]
Jul 11 03:42:35.501802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.501730    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 11 03:42:35.504132 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.504112    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 11 03:42:35.506983 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.506874    2582 manager.go:216] Machine: {Timestamp:2026-07-11 03:42:35.505482042 +0000 UTC m=+0.374928597 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102026 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d846c3615735c549767d7f9260d06 SystemUUID:ec2d846c-3615-735c-5497-67d7f9260d06 BootID:8ec8f2d4-e70a-4601-8a5f-83f68e9ef52d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cb:ce:a9:be:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cb:ce:a9:be:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:92:b0:78:10:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 11 03:42:35.506983 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.506978    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 11 03:42:35.507107 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.507063    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 11 03:42:35.510585 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.510560    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 11 03:42:35.510746 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.510589    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 11 03:42:35.511308 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.511298    2582 topology_manager.go:138] "Creating topology manager with none policy"
Jul 11 03:42:35.511344 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.511311    2582 container_manager_linux.go:309] "Creating device plugin manager"
Jul 11 03:42:35.511344 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.511319    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 11 03:42:35.511344 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.511333    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 11 03:42:35.512798 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.512788    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 11 03:42:35.515770 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.515758    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 11 03:42:35.515894 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.515884    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 11 03:42:35.515923 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.515915    2582 kubelet.go:486] "Attempting to sync node with API server"
Jul 11 03:42:35.515951 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.515930    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 11 03:42:35.515977 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.515954    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 11 03:42:35.515977 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.515971    2582 kubelet.go:398] "Adding apiserver pod source"
Jul 11 03:42:35.516039 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.516013    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 11 03:42:35.516439 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.516423    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 11 03:42:35.516508 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.516451    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 11 03:42:35.519207 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.519191    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 11 03:42:35.521319 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521306    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 11 03:42:35.521383 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521336    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 11 03:42:35.521580 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521570    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 11 03:42:35.521627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521585    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 11 03:42:35.521627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521592    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 11 03:42:35.521627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521597    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 11 03:42:35.521627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521604    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 11 03:42:35.521627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.521609    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 11 03:42:35.522186 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.522176    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 11 03:42:35.522229 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.522197    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 11 03:42:35.522229 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.522204    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 11 03:42:35.522229 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.522209    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 11 03:42:35.522229 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.522221    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 11 03:42:35.522229 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.522229    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 11 03:42:35.523275 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.523265    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 11 03:42:35.523275 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.523275    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 11 03:42:35.526984 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.526966    2582 server.go:1268] "Started kubelet"
Jul 11 03:42:35.527060 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.527012    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 11 03:42:35.527127 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.527052    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 11 03:42:35.527187 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.527150    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 11 03:42:35.527935 ip-10-0-140-230 systemd[1]: Started Kubernetes Kubelet.
Jul 11 03:42:35.528710 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.528528    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 11 03:42:35.529157 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.529043    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 11 03:42:35.529157 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.529117    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 11 03:42:35.529157 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.529130    2582 server.go:310] "Adding debug handlers to kubelet server"
Jul 11 03:42:35.529276 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.529129    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 11 03:42:35.533310 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.533270    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 11 03:42:35.534559 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.533477    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-230.ec2.internal.18c11f4feb5b4b7f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-230.ec2.internal,UID:ip-10-0-140-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-230.ec2.internal,},FirstTimestamp:2026-07-11 03:42:35.526933375 +0000 UTC m=+0.396379930,LastTimestamp:2026-07-11 03:42:35.526933375 +0000 UTC m=+0.396379930,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-230.ec2.internal,}"
Jul 11 03:42:35.536231 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.536216    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 11 03:42:35.536299 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.536233    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 11 03:42:35.536348 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.536292    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:35.537112 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.536957    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 11 03:42:35.537112 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.536974    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 11 03:42:35.537112 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.536955    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 11 03:42:35.537112 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537077    2582 reconstruct.go:97] "Volume reconstruction finished"
Jul 11 03:42:35.537112 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537084    2582 reconciler.go:29] "Reconciler: start to sync state"
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537321    2582 factory.go:153] Registering CRI-O factory
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537333    2582 factory.go:223] Registration of the crio container factory successfully
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537372    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537378    2582 factory.go:55] Registering systemd factory
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537385    2582 factory.go:223] Registration of the systemd container factory successfully
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537400    2582 factory.go:103] Registering Raw factory
Jul 11 03:42:35.537527 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.537427    2582 manager.go:1196] Started watching for new ooms in manager
Jul 11 03:42:35.538470 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.538378    2582 manager.go:349] Starting recovery of all containers
Jul 11 03:42:35.540302 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.540277    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 11 03:42:35.540400 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.540352    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 11 03:42:35.548621 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.548602    2582 manager.go:354] Recovery completed
Jul 11 03:42:35.550209 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.550181    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 11 03:42:35.552448 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.552429    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phtrj"
Jul 11 03:42:35.554697 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.554685    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.557253 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557231    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.557340 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557275    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.557340 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557287    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.557954 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557921    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 11 03:42:35.557954 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557938    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 11 03:42:35.557954 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557924    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.557954 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.557955    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 11 03:42:35.559289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.559277    2582 policy_none.go:49] "None policy: Start"
Jul 11 03:42:35.559338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.559294    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 11 03:42:35.559338 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.559225    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-230.ec2.internal.18c11f4fed29fcbd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-230.ec2.internal,UID:ip-10-0-140-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-230.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-230.ec2.internal,},FirstTimestamp:2026-07-11 03:42:35.557256381 +0000 UTC m=+0.426702943,LastTimestamp:2026-07-11 03:42:35.557256381 +0000 UTC m=+0.426702943,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-230.ec2.internal,}"
Jul 11 03:42:35.559338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.559305    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 11 03:42:35.560067 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.560048    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phtrj"
Jul 11 03:42:35.560145 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.560075    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.560145 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.560122    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.560145 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.560137    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.560449 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.560433    2582 policy_none.go:47] "Start"
Jul 11 03:42:35.594707 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.594686    2582 manager.go:337] "Starting Device Plugin manager"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.594721    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.594734    2582 server.go:87] "Starting device plugin registration server"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.595007    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.595019    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.595088    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.595190    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.595199    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.595658    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 11 03:42:35.602221 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.595717    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:35.637781 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.637740    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 11 03:42:35.639121 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.639078    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 11 03:42:35.639174 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.639141    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 11 03:42:35.639174 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.639163    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 11 03:42:35.639273 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.639257    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 11 03:42:35.641554 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.641531    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 11 03:42:35.695544 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.695496    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.696564 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.696535    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.696688 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.696575    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.696688 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.696585    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.696688 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.696614    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.705609 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.705560    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.705609 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.705583    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-230.ec2.internal\": node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:35.726017 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.725985    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:35.739383 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.739355    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"]
Jul 11 03:42:35.739444 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.739433    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.741203 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.741176    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.741315 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.741209    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.741315 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.741219    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.742434 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.742423    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.742578 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.742563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.742615 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.742594    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.743216 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.743197    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.743307 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.743228    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.743307 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.743238    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.743307 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.743200    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.743437 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.743311    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.743437 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.743323    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.744417 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.744404    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.744457 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.744430    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 11 03:42:35.745182 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.745164    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 11 03:42:35.745259 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.745199    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 11 03:42:35.745259 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.745214    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID"
Jul 11 03:42:35.760659 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.760634    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-230.ec2.internal\" not found" node="ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.764898 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.764872    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-230.ec2.internal\" not found" node="ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.826590 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.826562    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:35.838963 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.838938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049dea339b4ce50fc7bec48a2c1af6df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"049dea339b4ce50fc7bec48a2c1af6df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.839022 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.838969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049dea339b4ce50fc7bec48a2c1af6df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"049dea339b4ce50fc7bec48a2c1af6df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.839022 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.838989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32298df727bbd8ffb50d4d225716d8ff-config\") pod \"kube-apiserver-proxy-ip-10-0-140-230.ec2.internal\" (UID: \"32298df727bbd8ffb50d4d225716d8ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.927153 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:35.927116    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:35.939448 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.939426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049dea339b4ce50fc7bec48a2c1af6df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"049dea339b4ce50fc7bec48a2c1af6df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.939492 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.939462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049dea339b4ce50fc7bec48a2c1af6df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"049dea339b4ce50fc7bec48a2c1af6df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.939492 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.939481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32298df727bbd8ffb50d4d225716d8ff-config\") pod \"kube-apiserver-proxy-ip-10-0-140-230.ec2.internal\" (UID: \"32298df727bbd8ffb50d4d225716d8ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.939553 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.939527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/049dea339b4ce50fc7bec48a2c1af6df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"049dea339b4ce50fc7bec48a2c1af6df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.939605 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.939590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/049dea339b4ce50fc7bec48a2c1af6df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"049dea339b4ce50fc7bec48a2c1af6df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:35.939638 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:35.939631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32298df727bbd8ffb50d4d225716d8ff-config\") pod \"kube-apiserver-proxy-ip-10-0-140-230.ec2.internal\" (UID: \"32298df727bbd8ffb50d4d225716d8ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:36.027923 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.027846    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.062397 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.062374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:36.068378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.068359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:36.128175 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.128142    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.228687 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.228643    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.329342 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.329253    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.401363 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.401332    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 11 03:42:36.430245 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.430211    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.472693 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.472667    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 11 03:42:36.473324 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.472819    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 11 03:42:36.473324 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.472836    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 11 03:42:36.530307 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.530274    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.563670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.563631    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-10 03:37:36 +0000 UTC" deadline="2027-12-29 08:59:42.657718697 +0000 UTC"
Jul 11 03:42:36.563670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.563664    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12869h17m6.094057997s"
Jul 11 03:42:36.630997 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.630960    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.640388 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:36.640357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049dea339b4ce50fc7bec48a2c1af6df.slice/crio-6695f2327218301e7c90a45059a9a45abb97dd8c351ba8d3739f6a3a698aa758 WatchSource:0}: Error finding container 6695f2327218301e7c90a45059a9a45abb97dd8c351ba8d3739f6a3a698aa758: Status 404 returned error can't find the container with id 6695f2327218301e7c90a45059a9a45abb97dd8c351ba8d3739f6a3a698aa758
Jul 11 03:42:36.641146 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:36.641083    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32298df727bbd8ffb50d4d225716d8ff.slice/crio-a70f91954bc237ff8c9e6d18399786f59c557b7e73b5faa50d456eb6a5e41ef3 WatchSource:0}: Error finding container a70f91954bc237ff8c9e6d18399786f59c557b7e73b5faa50d456eb6a5e41ef3: Status 404 returned error can't find the container with id a70f91954bc237ff8c9e6d18399786f59c557b7e73b5faa50d456eb6a5e41ef3
Jul 11 03:42:36.645536 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.645516    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 03:42:36.654246 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:36.654226    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 11 03:42:36.731293 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.731238    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.831800 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.831766    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:36.932865 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:36.932830    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:37.033850 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.033807    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found"
Jul 11 03:42:37.054268 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.054244    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 11 03:42:37.136425 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.136391    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:37.147463 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.147433    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 11 03:42:37.148876 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.148854    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal"
Jul 11 03:42:37.160782 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.160755    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 11 03:42:37.517500 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.517467    2582 apiserver.go:52] "Watching apiserver"
Jul 11 03:42:37.527221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.527191    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 11 03:42:37.530853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.530434    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-hfzf5","kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs","openshift-image-registry/node-ca-nswmp","openshift-multus/multus-nlkqm","openshift-multus/network-metrics-daemon-qlm47","openshift-network-diagnostics/network-check-target-g28hp","openshift-ovn-kubernetes/ovnkube-node-8r9wt","kube-system/global-pull-secret-syncer-wgr8c","openshift-cluster-node-tuning-operator/tuned-2h4gs","openshift-dns/node-resolver-rdtx4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal","openshift-multus/multus-additional-cni-plugins-vlw6w","openshift-network-operator/iptables-alerter-qwqhv"]
Jul 11 03:42:37.533415 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.533396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:37.533524 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.533480    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:37.535919 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.535899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.537589 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.537462    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 11 03:42:37.538783 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.538765    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.539788 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.539766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 11 03:42:37.539788 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.539802    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 11 03:42:37.539932 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.539862    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.539977 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.539956    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 11 03:42:37.540113 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.540095    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sdmtl\""
Jul 11 03:42:37.540113 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.540111    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.542099 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.541280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.542973 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.542951    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vlc9j\""
Jul 11 03:42:37.543245 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.543226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.543557 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.543534    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.543787 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.543766    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 11 03:42:37.545062 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.544521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.545062 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.544613    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g6h7j\""
Jul 11 03:42:37.545062 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.544752    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.545062 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.544832    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 11 03:42:37.545062 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.545019    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.546084 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.546060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:37.546210 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.546163    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-registration-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-socket-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547377    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547417    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8sk7d\""
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6sl\" (UniqueName: \"kubernetes.io/projected/e0339b6f-0473-42e8-854f-c943ab81ec25-kube-api-access-wn6sl\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-device-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrw96\" (UniqueName: \"kubernetes.io/projected/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-kube-api-access-xrw96\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-system-cni-dir\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-cnibin\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-os-release\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.547690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.548670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-sys-fs\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.548670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.547684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.549333 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.549301    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 11 03:42:37.550271 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.550082    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.550271 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.550227    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:37.554238 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.554209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.556500 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.556481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 11 03:42:37.556610 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.556481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 11 03:42:37.556822 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.556796    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qxlvb\""
Jul 11 03:42:37.557238 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.557146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.559147 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.559099    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.559687 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.559665    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 11 03:42:37.559813 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.559767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t58cd\""
Jul 11 03:42:37.559813 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.559670    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.559946 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.559934    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 11 03:42:37.560260 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.560166    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.562556 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.562528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nck22\""
Jul 11 03:42:37.562779 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.562760    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.562978 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.562966    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.565894 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.565836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.568413 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.568221    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sz6cw\""
Jul 11 03:42:37.568413 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.568360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.568745 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.568728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.569424 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.569366    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.570619 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.570602    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9dbs"
Jul 11 03:42:37.571889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.571161    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-trblz\""
Jul 11 03:42:37.571889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.571444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 11 03:42:37.571889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.571648    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 11 03:42:37.572105 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.572095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 11 03:42:37.572369 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.572261    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 11 03:42:37.573279 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.573257    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 11 03:42:37.574196 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.574160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 11 03:42:37.579406 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.579318    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9dbs"
Jul 11 03:42:37.638309 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.638282    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 11 03:42:37.645098 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.645043    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" event={"ID":"32298df727bbd8ffb50d4d225716d8ff","Type":"ContainerStarted","Data":"a70f91954bc237ff8c9e6d18399786f59c557b7e73b5faa50d456eb6a5e41ef3"}
Jul 11 03:42:37.646773 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.646723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" event={"ID":"049dea339b4ce50fc7bec48a2c1af6df","Type":"ContainerStarted","Data":"6695f2327218301e7c90a45059a9a45abb97dd8c351ba8d3739f6a3a698aa758"}
Jul 11 03:42:37.648709 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.648808 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-conf-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.648808 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-run-netns\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.648808 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.648959 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmvv\" (UniqueName: \"kubernetes.io/projected/616ac4cd-27d1-4d45-b171-65cd42924152-kube-api-access-cpmvv\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.648959 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-registration-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.648959 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-k8s-cni-cncf-io\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.648959 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-lib-modules\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.648959 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21e8f34f-1131-4a4e-9830-7eba4d3135e4-hosts-file\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.648959 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-registration-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.648964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-tuned\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-etc-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb2d92a6-4c87-47ae-af3f-8634e40903e6-serviceca\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-socket-dir-parent\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aeb18047-4145-49e9-844e-4bdb4f0f8eb0-agent-certs\") pod \"konnectivity-agent-hfzf5\" (UID: \"aeb18047-4145-49e9-844e-4bdb4f0f8eb0\") " pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-host\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.649266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5jz\" (UniqueName: \"kubernetes.io/projected/4e522d6f-2360-44a9-8d75-6013ab33d709-kube-api-access-lm5jz\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2d92a6-4c87-47ae-af3f-8634e40903e6-host\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/616ac4cd-27d1-4d45-b171-65cd42924152-host-slash\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-cnibin\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e41e790-86e5-4643-8afc-2f49245db72e-kubelet-config\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e41e790-86e5-4643-8afc-2f49245db72e-dbus\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-os-release\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-multus-certs\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysctl-d\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-ovnkube-script-lib\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/616ac4cd-27d1-4d45-b171-65cd42924152-iptables-alerter-script\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-socket-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.649690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-cni-binary-copy\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-daemon-config\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-etc-kubernetes\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aeb18047-4145-49e9-844e-4bdb4f0f8eb0-konnectivity-ca\") pod \"konnectivity-agent-hfzf5\" (UID: \"aeb18047-4145-49e9-844e-4bdb4f0f8eb0\") " pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-run\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-systemd-units\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-device-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrw96\" (UniqueName: \"kubernetes.io/projected/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-kube-api-access-xrw96\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-var-lib-kubelet\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.649992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-slash\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-systemd\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-run-ovn-kubernetes\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-env-overrides\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmwf\" (UniqueName: \"kubernetes.io/projected/eb2d92a6-4c87-47ae-af3f-8634e40903e6-kube-api-access-grmwf\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-sys-fs\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysconfig\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.650287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-sys\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-var-lib-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-cni-netd\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-kubernetes\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-device-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e522d6f-2360-44a9-8d75-6013ab33d709-ovn-node-metrics-cert\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdpw\" (UniqueName: \"kubernetes.io/projected/2be65b1e-0556-42b8-818d-e596d95d4a15-kube-api-access-jxdpw\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-netns\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-cni-bin\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-hostroot\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6sl\" (UniqueName: \"kubernetes.io/projected/e0339b6f-0473-42e8-854f-c943ab81ec25-kube-api-access-wn6sl\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-modprobe-d\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysctl-conf\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e8f34f-1131-4a4e-9830-7eba4d3135e4-tmp-dir\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-ovn\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-cni-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-cnibin\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-system-cni-dir\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-os-release\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-os-release\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-socket-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-cnibin\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.650876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ppg\" (UniqueName: \"kubernetes.io/projected/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-kube-api-access-r4ppg\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afd1770e-98fb-4c50-9a22-6337f1e1439a-tmp\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-cni-multus\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-kubelet\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-systemd\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-sys-fs\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-cni-bin\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.651869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0339b6f-0473-42e8-854f-c943ab81ec25-system-cni-dir\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-system-cni-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-log-socket\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.651969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.652031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqmv\" (UniqueName: \"kubernetes.io/projected/afd1770e-98fb-4c50-9a22-6337f1e1439a-kube-api-access-9tqmv\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.652063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6dj\" (UniqueName: \"kubernetes.io/projected/21e8f34f-1131-4a4e-9830-7eba4d3135e4-kube-api-access-vl6dj\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.652097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-kubelet\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.652144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-node-log\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.652195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-ovnkube-config\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.652802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.652339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0339b6f-0473-42e8-854f-c943ab81ec25-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.661528 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.661489    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 11 03:42:37.662495 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.662476    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:42:37.662495 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.662496    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:42:37.662615 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.662514    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:37.662615 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.662570    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:38.162546641 +0000 UTC m=+3.031993186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:37.667497 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.667289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrw96\" (UniqueName: \"kubernetes.io/projected/45b45787-7cc4-4a0e-9b96-a1ae795a06d9-kube-api-access-xrw96\") pod \"aws-ebs-csi-driver-node-fhrrs\" (UID: \"45b45787-7cc4-4a0e-9b96-a1ae795a06d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.667497 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.667445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6sl\" (UniqueName: \"kubernetes.io/projected/e0339b6f-0473-42e8-854f-c943ab81ec25-kube-api-access-wn6sl\") pod \"multus-additional-cni-plugins-vlw6w\" (UID: \"e0339b6f-0473-42e8-854f-c943ab81ec25\") " pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.753204 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-modprobe-d\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.753380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysctl-conf\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.753380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e8f34f-1131-4a4e-9830-7eba4d3135e4-tmp-dir\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.753380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-ovn\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.753380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-cni-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-cnibin\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-modprobe-d\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ppg\" (UniqueName: \"kubernetes.io/projected/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-kube-api-access-r4ppg\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afd1770e-98fb-4c50-9a22-6337f1e1439a-tmp\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-cni-multus\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-cnibin\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysctl-conf\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-kubelet\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-cni-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-systemd\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-cni-bin\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-cni-multus\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-ovn\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-system-cni-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-log-socket\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.753679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-systemd\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-cni-bin\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqmv\" (UniqueName: \"kubernetes.io/projected/afd1770e-98fb-4c50-9a22-6337f1e1439a-kube-api-access-9tqmv\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-system-cni-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-log-socket\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6dj\" (UniqueName: \"kubernetes.io/projected/21e8f34f-1131-4a4e-9830-7eba4d3135e4-kube-api-access-vl6dj\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-kubelet\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-kubelet\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-node-log\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-ovnkube-config\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-conf-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-run-netns\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.753977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmvv\" (UniqueName: \"kubernetes.io/projected/616ac4cd-27d1-4d45-b171-65cd42924152-kube-api-access-cpmvv\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-k8s-cni-cncf-io\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-lib-modules\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.754652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21e8f34f-1131-4a4e-9830-7eba4d3135e4-hosts-file\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-tuned\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-run-netns\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-etc-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb2d92a6-4c87-47ae-af3f-8634e40903e6-serviceca\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-etc-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-conf-dir\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-k8s-cni-cncf-io\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21e8f34f-1131-4a4e-9830-7eba4d3135e4-tmp-dir\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21e8f34f-1131-4a4e-9830-7eba4d3135e4-hosts-file\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-kubelet\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754457    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-node-log\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.754485    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-lib-modules\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.754576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:38.254555547 +0000 UTC m=+3.124002089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-ovnkube-config\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.754725    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:37.755521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.754774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:37.756419 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:37.755085    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:38.255041904 +0000 UTC m=+3.124488454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.759388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb2d92a6-4c87-47ae-af3f-8634e40903e6-serviceca\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.759433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-tuned\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.759571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afd1770e-98fb-4c50-9a22-6337f1e1439a-tmp\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.759945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-socket-dir-parent\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.759985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aeb18047-4145-49e9-844e-4bdb4f0f8eb0-agent-certs\") pod \"konnectivity-agent-hfzf5\" (UID: \"aeb18047-4145-49e9-844e-4bdb4f0f8eb0\") " pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-host\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5jz\" (UniqueName: \"kubernetes.io/projected/4e522d6f-2360-44a9-8d75-6013ab33d709-kube-api-access-lm5jz\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2d92a6-4c87-47ae-af3f-8634e40903e6-host\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/616ac4cd-27d1-4d45-b171-65cd42924152-host-slash\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e41e790-86e5-4643-8afc-2f49245db72e-kubelet-config\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e41e790-86e5-4643-8afc-2f49245db72e-dbus\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-os-release\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-multus-certs\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysctl-d\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-ovnkube-script-lib\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/616ac4cd-27d1-4d45-b171-65cd42924152-iptables-alerter-script\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.764326 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-cni-binary-copy\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-daemon-config\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-etc-kubernetes\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aeb18047-4145-49e9-844e-4bdb4f0f8eb0-konnectivity-ca\") pod \"konnectivity-agent-hfzf5\" (UID: \"aeb18047-4145-49e9-844e-4bdb4f0f8eb0\") " pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-run\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-systemd-units\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-var-lib-kubelet\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-slash\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-systemd\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-run-ovn-kubernetes\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-env-overrides\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmwf\" (UniqueName: \"kubernetes.io/projected/eb2d92a6-4c87-47ae-af3f-8634e40903e6-kube-api-access-grmwf\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysconfig\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-sys\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-var-lib-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-cni-netd\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-kubernetes\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.765289 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e522d6f-2360-44a9-8d75-6013ab33d709-ovn-node-metrics-cert\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdpw\" (UniqueName: \"kubernetes.io/projected/2be65b1e-0556-42b8-818d-e596d95d4a15-kube-api-access-jxdpw\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-netns\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-cni-bin\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-hostroot\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.760848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-hostroot\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-socket-dir-parent\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-run\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-multus-certs\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysconfig\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-host\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-sysctl-d\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-systemd-units\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e41e790-86e5-4643-8afc-2f49245db72e-kubelet-config\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e41e790-86e5-4643-8afc-2f49245db72e-dbus\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-var-lib-kubelet\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-os-release\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.766151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/616ac4cd-27d1-4d45-b171-65cd42924152-host-slash\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761917    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-etc-kubernetes\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-var-lib-openvswitch\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.761962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-slash\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.762033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-cni-netd\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.762030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afd1770e-98fb-4c50-9a22-6337f1e1439a-sys\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.762462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2d92a6-4c87-47ae-af3f-8634e40903e6-host\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.762473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-env-overrides\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.762478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e522d6f-2360-44a9-8d75-6013ab33d709-ovnkube-script-lib\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/616ac4cd-27d1-4d45-b171-65cd42924152-iptables-alerter-script\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-run-systemd\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e522d6f-2360-44a9-8d75-6013ab33d709-host-run-ovn-kubernetes\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-etc-kubernetes\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-run-netns\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-cni-binary-copy\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.763901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-host-var-lib-cni-bin\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.764159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aeb18047-4145-49e9-844e-4bdb4f0f8eb0-agent-certs\") pod \"konnectivity-agent-hfzf5\" (UID: \"aeb18047-4145-49e9-844e-4bdb4f0f8eb0\") " pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.764159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-multus-daemon-config\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.767110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.764841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aeb18047-4145-49e9-844e-4bdb4f0f8eb0-konnectivity-ca\") pod \"konnectivity-agent-hfzf5\" (UID: \"aeb18047-4145-49e9-844e-4bdb4f0f8eb0\") " pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.768052 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.766522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6dj\" (UniqueName: \"kubernetes.io/projected/21e8f34f-1131-4a4e-9830-7eba4d3135e4-kube-api-access-vl6dj\") pod \"node-resolver-rdtx4\" (UID: \"21e8f34f-1131-4a4e-9830-7eba4d3135e4\") " pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.768052 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.767257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqmv\" (UniqueName: \"kubernetes.io/projected/afd1770e-98fb-4c50-9a22-6337f1e1439a-kube-api-access-9tqmv\") pod \"tuned-2h4gs\" (UID: \"afd1770e-98fb-4c50-9a22-6337f1e1439a\") " pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.768052 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.767407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e522d6f-2360-44a9-8d75-6013ab33d709-ovn-node-metrics-cert\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.768357 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.768306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ppg\" (UniqueName: \"kubernetes.io/projected/7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c-kube-api-access-r4ppg\") pod \"multus-nlkqm\" (UID: \"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c\") " pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.769675 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.769648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmwf\" (UniqueName: \"kubernetes.io/projected/eb2d92a6-4c87-47ae-af3f-8634e40903e6-kube-api-access-grmwf\") pod \"node-ca-nswmp\" (UID: \"eb2d92a6-4c87-47ae-af3f-8634e40903e6\") " pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.770281 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.770260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmvv\" (UniqueName: \"kubernetes.io/projected/616ac4cd-27d1-4d45-b171-65cd42924152-kube-api-access-cpmvv\") pod \"iptables-alerter-qwqhv\" (UID: \"616ac4cd-27d1-4d45-b171-65cd42924152\") " pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.770481 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.770462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5jz\" (UniqueName: \"kubernetes.io/projected/4e522d6f-2360-44a9-8d75-6013ab33d709-kube-api-access-lm5jz\") pod \"ovnkube-node-8r9wt\" (UID: \"4e522d6f-2360-44a9-8d75-6013ab33d709\") " pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:37.773148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.773124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdpw\" (UniqueName: \"kubernetes.io/projected/2be65b1e-0556-42b8-818d-e596d95d4a15-kube-api-access-jxdpw\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:37.854562 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.854530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlw6w"
Jul 11 03:42:37.864725 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.864701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs"
Jul 11 03:42:37.876728 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.876693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nswmp"
Jul 11 03:42:37.884875 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.884853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nlkqm"
Jul 11 03:42:37.893811 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.893791    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:42:37.904679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.904650    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qwqhv"
Jul 11 03:42:37.913591 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.913567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2h4gs"
Jul 11 03:42:37.921535 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.921510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rdtx4"
Jul 11 03:42:37.927563 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:37.927537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:42:38.164083 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:38.164005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:38.164284 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.164189    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:42:38.164284 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.164209    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:42:38.164284 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.164220    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:38.164398 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.164288    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:39.164267065 +0000 UTC m=+4.033713612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:38.169327 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:38.169291    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 11 03:42:38.265233 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:38.265198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:38.265421 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:38.265242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:38.265421 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.265371    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:38.265421 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.265373    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:38.265553 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.265429    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:39.26541149 +0000 UTC m=+4.134858036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:38.265553 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:38.265443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:39.265437078 +0000 UTC m=+4.134883622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:38.583691 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:38.583568    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-10 03:37:38 +0000 UTC" deadline="2028-02-13 19:27:22.242629291 +0000 UTC"
Jul 11 03:42:38.583691 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:38.583648    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13983h44m43.658991143s"
Jul 11 03:42:38.752961 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:38.752932    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b45787_7cc4_4a0e_9b96_a1ae795a06d9.slice/crio-79071b428dd0e5d84ce84cbcb839e5d901cba1114f33856b5801c6d4a4053c4a WatchSource:0}: Error finding container 79071b428dd0e5d84ce84cbcb839e5d901cba1114f33856b5801c6d4a4053c4a: Status 404 returned error can't find the container with id 79071b428dd0e5d84ce84cbcb839e5d901cba1114f33856b5801c6d4a4053c4a
Jul 11 03:42:38.754639 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:38.754608    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e8f34f_1131_4a4e_9830_7eba4d3135e4.slice/crio-0518e47f4bfbe9aff34f3924eeb394d4f727c8a8f01f02914bbeed25bcfaace0 WatchSource:0}: Error finding container 0518e47f4bfbe9aff34f3924eeb394d4f727c8a8f01f02914bbeed25bcfaace0: Status 404 returned error can't find the container with id 0518e47f4bfbe9aff34f3924eeb394d4f727c8a8f01f02914bbeed25bcfaace0
Jul 11 03:42:38.780736 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:38.777048    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e522d6f_2360_44a9_8d75_6013ab33d709.slice/crio-af479d66e215f1d5eedfc8bce3948574d84f306f67d51bcba14e736510417332 WatchSource:0}: Error finding container af479d66e215f1d5eedfc8bce3948574d84f306f67d51bcba14e736510417332: Status 404 returned error can't find the container with id af479d66e215f1d5eedfc8bce3948574d84f306f67d51bcba14e736510417332
Jul 11 03:42:38.780736 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:38.780690    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3fb74a_3bb1_45d1_94f4_c52eb1c5074c.slice/crio-c16dc9de11ecde273b80386e90259be8f17528efa6559fa0f590d0131ef98191 WatchSource:0}: Error finding container c16dc9de11ecde273b80386e90259be8f17528efa6559fa0f590d0131ef98191: Status 404 returned error can't find the container with id c16dc9de11ecde273b80386e90259be8f17528efa6559fa0f590d0131ef98191
Jul 11 03:42:38.781686 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:38.781664    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616ac4cd_27d1_4d45_b171_65cd42924152.slice/crio-1a42c7e883936195402bf628c7dfb9d572db6204802b851a2eb4efe59c3f3fe5 WatchSource:0}: Error finding container 1a42c7e883936195402bf628c7dfb9d572db6204802b851a2eb4efe59c3f3fe5: Status 404 returned error can't find the container with id 1a42c7e883936195402bf628c7dfb9d572db6204802b851a2eb4efe59c3f3fe5
Jul 11 03:42:38.782870 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:42:38.782842    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb18047_4145_49e9_844e_4bdb4f0f8eb0.slice/crio-67a66f739c562d078f8bbec93ff10d58339b2d43e7c6539fe7f583f0e1aa27c1 WatchSource:0}: Error finding container 67a66f739c562d078f8bbec93ff10d58339b2d43e7c6539fe7f583f0e1aa27c1: Status 404 returned error can't find the container with id 67a66f739c562d078f8bbec93ff10d58339b2d43e7c6539fe7f583f0e1aa27c1
Jul 11 03:42:39.172484 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.172177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:39.172627 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.172386    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:42:39.172627 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.172523    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:42:39.172627 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.172538    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:39.172627 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.172598    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:41.172580372 +0000 UTC m=+6.042026930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:39.274377 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.273542    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:39.274377 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.273653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:39.274377 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.273780    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:39.274377 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.273848    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:41.273826946 +0000 UTC m=+6.143273500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:39.274377 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.273936    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:39.274377 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.273976    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:41.273962826 +0000 UTC m=+6.143409373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:39.584607 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.584451    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-10 03:37:38 +0000 UTC" deadline="2027-12-14 18:32:37.925311524 +0000 UTC"
Jul 11 03:42:39.584607 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.584493    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12518h49m58.340823841s"
Jul 11 03:42:39.643393 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.642472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:39.643393 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.642615    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:39.643393 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.643031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:39.643393 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.643153    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:39.643393 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.643257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:39.643393 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:39.643335    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:39.665561 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.665530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" event={"ID":"32298df727bbd8ffb50d4d225716d8ff","Type":"ContainerStarted","Data":"78479e098532cf91e00f0915625331e6ba5dbcc8ad4942cf37ec47691b543893"}
Jul 11 03:42:39.672639 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.672603    2582 generic.go:356] "Generic (PLEG): container finished" podID="049dea339b4ce50fc7bec48a2c1af6df" containerID="ae4a6ee6d376283ea657ad0d86e45b30ea7b1f486a6efc23fe945a52a551f9df" exitCode=0
Jul 11 03:42:39.672800 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.672696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" event={"ID":"049dea339b4ce50fc7bec48a2c1af6df","Type":"ContainerDied","Data":"ae4a6ee6d376283ea657ad0d86e45b30ea7b1f486a6efc23fe945a52a551f9df"}
Jul 11 03:42:39.677595 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.677566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2h4gs" event={"ID":"afd1770e-98fb-4c50-9a22-6337f1e1439a","Type":"ContainerStarted","Data":"421c814c999291a3feb6a15b2d89401ffef2e0bb9f14c9faa4bd3803cc6e584a"}
Jul 11 03:42:39.679635 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.679550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerStarted","Data":"d2e19488983664349dbbd912709d04275a01f7ea557d7d1517f7c08920c3bbca"}
Jul 11 03:42:39.687815 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.687780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qwqhv" event={"ID":"616ac4cd-27d1-4d45-b171-65cd42924152","Type":"ContainerStarted","Data":"1a42c7e883936195402bf628c7dfb9d572db6204802b851a2eb4efe59c3f3fe5"}
Jul 11 03:42:39.691975 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.691947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nlkqm" event={"ID":"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c","Type":"ContainerStarted","Data":"c16dc9de11ecde273b80386e90259be8f17528efa6559fa0f590d0131ef98191"}
Jul 11 03:42:39.699054 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.699001    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" podStartSLOduration=2.698980457 podStartE2EDuration="2.698980457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:39.680720063 +0000 UTC m=+4.550166632" watchObservedRunningTime="2026-07-11 03:42:39.698980457 +0000 UTC m=+4.568427026"
Jul 11 03:42:39.704391 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.704255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"af479d66e215f1d5eedfc8bce3948574d84f306f67d51bcba14e736510417332"}
Jul 11 03:42:39.707667 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.707605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rdtx4" event={"ID":"21e8f34f-1131-4a4e-9830-7eba4d3135e4","Type":"ContainerStarted","Data":"0518e47f4bfbe9aff34f3924eeb394d4f727c8a8f01f02914bbeed25bcfaace0"}
Jul 11 03:42:39.715037 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.714956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nswmp" event={"ID":"eb2d92a6-4c87-47ae-af3f-8634e40903e6","Type":"ContainerStarted","Data":"cbd0dc696428bcc5426fac3bcc53d52aba0d278bf7c75a506736c66a4a576e18"}
Jul 11 03:42:39.732743 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.732703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hfzf5" event={"ID":"aeb18047-4145-49e9-844e-4bdb4f0f8eb0","Type":"ContainerStarted","Data":"67a66f739c562d078f8bbec93ff10d58339b2d43e7c6539fe7f583f0e1aa27c1"}
Jul 11 03:42:39.756906 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:39.756850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs" event={"ID":"45b45787-7cc4-4a0e-9b96-a1ae795a06d9","Type":"ContainerStarted","Data":"79071b428dd0e5d84ce84cbcb839e5d901cba1114f33856b5801c6d4a4053c4a"}
Jul 11 03:42:40.780235 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:40.780040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" event={"ID":"049dea339b4ce50fc7bec48a2c1af6df","Type":"ContainerStarted","Data":"4ff2c00142d056d1eefe3aeb69d7a975cace2da86c95a18717955cb4d3956daa"}
Jul 11 03:42:40.796358 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:40.795900    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" podStartSLOduration=3.795876965 podStartE2EDuration="3.795876965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:40.794965046 +0000 UTC m=+5.664411614" watchObservedRunningTime="2026-07-11 03:42:40.795876965 +0000 UTC m=+5.665323533"
Jul 11 03:42:41.192896 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:41.192799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:41.193042 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.193008    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:42:41.193042 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.193032    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:42:41.193158 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.193046    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:41.193158 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.193107    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:45.193087059 +0000 UTC m=+10.062533619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:41.294878 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:41.293965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:41.294878 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:41.294069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:41.294878 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.294213    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:41.294878 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.294279    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:45.294260083 +0000 UTC m=+10.163706633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:41.294878 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.294710    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:41.294878 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.294761    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:45.294744818 +0000 UTC m=+10.164191365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:41.640492 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:41.640401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:41.640682 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.640546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:41.641069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:41.641046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:41.641251 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.641210    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:41.641580 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:41.641449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:41.641580 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:41.641546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:43.641091 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:43.640282    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:43.641091 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:43.640421    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:43.641091 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:43.640816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:43.641091 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:43.640905    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:43.641091 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:43.640974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:43.641091 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:43.641051    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:45.230258 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:45.230215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:45.230923 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.230748    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:42:45.230923 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.230777    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:42:45.230923 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.230802    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:45.230923 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.230870    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:53.230843913 +0000 UTC m=+18.100290474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:45.331789 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:45.331708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:45.331789 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:45.331763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:45.332143 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.331948    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:45.332143 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.332014    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:53.331995773 +0000 UTC m=+18.201442335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:45.332442 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.332423    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:45.332519 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.332473    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:42:53.332458811 +0000 UTC m=+18.201905376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:45.640797 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:45.640714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:45.640947 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.640810    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:45.641452 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:45.641173    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:45.641452 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.641283    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:45.641452 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:45.641351    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:45.641452 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:45.641417    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:47.641906 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:47.641872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:47.642373 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:47.642018    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:47.642422 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:47.642393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:47.642506 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:47.642486    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:47.642556 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:47.642537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:47.642619 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:47.642600    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:49.642908 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:49.642877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:49.642908 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:49.642900    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:49.643547 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:49.643009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:49.643547 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:49.643065    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:49.643547 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:49.643122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:49.643547 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:49.643175    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:51.640189 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:51.640153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:51.640189 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:51.640184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:51.640670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:51.640159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:51.640670 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:51.640275    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:51.640670 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:51.640369    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:51.640670 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:51.640483    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:53.295361 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:53.295325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:53.295737 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.295519    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:42:53.295737 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.295547    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:42:53.295737 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.295563    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:53.295737 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.295625    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:09.295606521 +0000 UTC m=+34.165053069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:42:53.395698 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:53.395666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:53.395859 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:53.395703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:53.395859 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.395816    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:53.395859 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.395817    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:53.396026 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.395876    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:09.395861781 +0000 UTC m=+34.265308322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:42:53.396026 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.395896    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:09.395885078 +0000 UTC m=+34.265331621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:42:53.639473 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:53.639393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:53.639473 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:53.639432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:53.639679 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:53.639394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:53.639679 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.639511    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:53.639679 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.639590    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:53.639820 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:53.639672    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:55.640186 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:55.640146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:55.640186 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:55.640166    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:55.640709 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:55.640253    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:55.640709 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:55.640305    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:55.640709 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:55.640344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:55.640709 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:55.640438    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:57.639824 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:57.639790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:57.640298 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:57.639787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:57.640298 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:57.639908    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:57.640298 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:57.639938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:57.640298 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:57.640037    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:57.640298 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:57.640166    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:58.817928 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.817773    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:42:58.818524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.818280    2582 generic.go:356] "Generic (PLEG): container finished" podID="4e522d6f-2360-44a9-8d75-6013ab33d709" containerID="146729e8e3b1304952e67a4291d65258dafd3a32cc3376c349452e9aeac737b9" exitCode=1
Jul 11 03:42:58.818524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.818346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"a8fb46592a246c526b144398e3381442534bba7d7283064638493e678e81d387"}
Jul 11 03:42:58.818524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.818412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"522053045e90ead5e2df340bd630c5ddb4a0c35e88e032b1344900507e16c995"}
Jul 11 03:42:58.818524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.818442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"8573d5ca80608a7373b05646970713df45739cde002e767ba8d95d8622bb369e"}
Jul 11 03:42:58.818524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.818479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerDied","Data":"146729e8e3b1304952e67a4291d65258dafd3a32cc3376c349452e9aeac737b9"}
Jul 11 03:42:58.818524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.818509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"c72ecb4ee5f9d732aef0b446b737f5041c4c38a58e92f37b6348e7d2fcf5ede4"}
Jul 11 03:42:58.825613 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.825283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rdtx4" event={"ID":"21e8f34f-1131-4a4e-9830-7eba4d3135e4","Type":"ContainerStarted","Data":"4038aa970fb71de882a4cc586d4fe0f9384e45936418606a0a0abf8a16369c96"}
Jul 11 03:42:58.829036 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.828993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nswmp" event={"ID":"eb2d92a6-4c87-47ae-af3f-8634e40903e6","Type":"ContainerStarted","Data":"74c979837553acc7b9be34df2b35ce5204110ec91ba939f79a53781697af9359"}
Jul 11 03:42:58.831167 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.831142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hfzf5" event={"ID":"aeb18047-4145-49e9-844e-4bdb4f0f8eb0","Type":"ContainerStarted","Data":"73b02952eae1323e3577a0d2424e74969d42c195c62dfe2c6d264fbc007a81a1"}
Jul 11 03:42:58.833146 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.833064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs" event={"ID":"45b45787-7cc4-4a0e-9b96-a1ae795a06d9","Type":"ContainerStarted","Data":"4dfbf4cd29ad6c7ebaed944f49a61d32d861097e7fad5477640fe06b31cc4def"}
Jul 11 03:42:58.834886 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.834864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2h4gs" event={"ID":"afd1770e-98fb-4c50-9a22-6337f1e1439a","Type":"ContainerStarted","Data":"b92a62be6bb7e5fa3cd97c073277395ad8a8476c2eff9704619d0d286a4c4ef8"}
Jul 11 03:42:58.836656 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.836635    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0339b6f-0473-42e8-854f-c943ab81ec25" containerID="82ed32271ec025eb59df423506f39a36404ab062c0bf5195fc6c78fc7ba3db44" exitCode=0
Jul 11 03:42:58.836720 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.836691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerDied","Data":"82ed32271ec025eb59df423506f39a36404ab062c0bf5195fc6c78fc7ba3db44"}
Jul 11 03:42:58.840742 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.840721    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nlkqm" event={"ID":"7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c","Type":"ContainerStarted","Data":"0d7be0ed5f880a2c50c811e91e7fe5f670344863658412a0191ec878407dbdff"}
Jul 11 03:42:58.840870 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.840826    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rdtx4" podStartSLOduration=3.54836247 podStartE2EDuration="22.840809819s" totalImagesPullingTime="19.292447349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:58.839890015 +0000 UTC m=+23.709336581" watchObservedRunningTime="2026-07-11 03:42:58.840809819 +0000 UTC m=+23.710256384"
Jul 11 03:42:58.879340 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.879146    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2h4gs" podStartSLOduration=3.582655196 podStartE2EDuration="22.879126941s" totalImagesPullingTime="19.296471745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:58.857640232 +0000 UTC m=+23.727086799" watchObservedRunningTime="2026-07-11 03:42:58.879126941 +0000 UTC m=+23.748573504"
Jul 11 03:42:58.894166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.894086    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hfzf5" podStartSLOduration=8.236266563 podStartE2EDuration="22.894064688s" totalImagesPullingTime="14.657798125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:58.89355674 +0000 UTC m=+23.763003305" watchObservedRunningTime="2026-07-11 03:42:58.894064688 +0000 UTC m=+23.763511254"
Jul 11 03:42:58.907450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.907399    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nswmp" podStartSLOduration=3.612507422 podStartE2EDuration="22.907371317s" totalImagesPullingTime="19.294863895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:58.906701453 +0000 UTC m=+23.776148021" watchObservedRunningTime="2026-07-11 03:42:58.907371317 +0000 UTC m=+23.776817885"
Jul 11 03:42:58.923331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:58.923289    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nlkqm" podStartSLOduration=3.596756384 podStartE2EDuration="22.923273622s" totalImagesPullingTime="19.326517238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:58.922774025 +0000 UTC m=+23.792220590" watchObservedRunningTime="2026-07-11 03:42:58.923273622 +0000 UTC m=+23.792720186"
Jul 11 03:42:59.640672 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.640469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:42:59.640854 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.640478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:42:59.640854 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:59.640774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:42:59.640854 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.640512    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:42:59.641011 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:59.640865    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:42:59.641011 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:42:59.640934    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:42:59.778581 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.778546    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 11 03:42:59.844629 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.844589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qwqhv" event={"ID":"616ac4cd-27d1-4d45-b171-65cd42924152","Type":"ContainerStarted","Data":"4c0f7c7014f41ba2a17aeb999f5c9308bf4c6602149987b7b0640dc71baa7e8c"}
Jul 11 03:42:59.847997 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.847969    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:42:59.848496 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.848462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"9a51e99644d431fcf396c94369a8b744595c1a2d2c06e583075dc8a6e68c2a6d"}
Jul 11 03:42:59.850435 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.850415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs" event={"ID":"45b45787-7cc4-4a0e-9b96-a1ae795a06d9","Type":"ContainerStarted","Data":"8cd5c393fde4b0fb0bf591f05e51970100653afc2d39c67a2768a1bd492652c8"}
Jul 11 03:42:59.858518 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:42:59.858438    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qwqhv" podStartSLOduration=9.198260098 podStartE2EDuration="23.858420979s" totalImagesPullingTime="14.660160881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:42:59.858001359 +0000 UTC m=+24.727447923" watchObservedRunningTime="2026-07-11 03:42:59.858420979 +0000 UTC m=+24.727867547"
Jul 11 03:43:00.609695 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:00.609611    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-11T03:42:59.778575319Z","UUID":"d6bf4f17-ae91-44b6-af17-d0f67cb8ae14","Handler":null,"Name":"","Endpoint":""}
Jul 11 03:43:00.611471 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:00.611451    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 11 03:43:00.611593 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:00.611486    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 11 03:43:00.855392 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:00.855295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs" event={"ID":"45b45787-7cc4-4a0e-9b96-a1ae795a06d9","Type":"ContainerStarted","Data":"edaec485842836c31308134e0c3acc74f5d323037210d04a0aa0e67361200fc0"}
Jul 11 03:43:00.871422 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:00.871359    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhrrs" podStartSLOduration=3.526112496 podStartE2EDuration="24.871345011s" totalImagesPullingTime="21.345232515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:00.870758629 +0000 UTC m=+25.740205194" watchObservedRunningTime="2026-07-11 03:43:00.871345011 +0000 UTC m=+25.740791574"
Jul 11 03:43:01.640500 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.640466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:01.640691 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.640513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:01.640691 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.640469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:01.640691 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:01.640599    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:43:01.640856 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:01.640708    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:43:01.640856 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:01.640803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:43:01.861070 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.861039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:43:01.861564 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.861531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"7899fce94e7dffcbe5c6d1a65a7bbf55ca3a6ec62571796e4994670757d0297c"}
Jul 11 03:43:01.961495 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.961422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:43:01.962374 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:01.962354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:43:02.864338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:02.864312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:43:02.864958 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:02.864936    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hfzf5"
Jul 11 03:43:03.639880 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.639616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:03.640032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.639616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:03.640032 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:03.640000    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:43:03.640206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.639645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:03.640206 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:03.639883    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:43:03.640206 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:03.640128    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:43:03.867352 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.867315    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0339b6f-0473-42e8-854f-c943ab81ec25" containerID="4bce8dc4b99377cd20f0aad04c508d306c3b6869c1f84cdb14a9f8ff831044d2" exitCode=0
Jul 11 03:43:03.867841 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.867413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerDied","Data":"4bce8dc4b99377cd20f0aad04c508d306c3b6869c1f84cdb14a9f8ff831044d2"}
Jul 11 03:43:03.871046 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.871025    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:43:03.871514 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.871489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"95951a14a309302dad351156c84be57b48a7872722c76e289c87b5e9dbdab1c6"}
Jul 11 03:43:03.871702 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.871681    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:43:03.871771 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.871754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:43:03.871831 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.871795    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:43:03.871868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.871854    2582 scope.go:117] "RemoveContainer" containerID="146729e8e3b1304952e67a4291d65258dafd3a32cc3376c349452e9aeac737b9"
Jul 11 03:43:03.888532 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.888512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:43:03.888669 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:03.888656    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:43:04.876668 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:04.876457    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:43:04.877057 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:04.876995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" event={"ID":"4e522d6f-2360-44a9-8d75-6013ab33d709","Type":"ContainerStarted","Data":"6924cb0e236b212bff333b6e88112a299cbc9a1b85bff4398f6f5875f771f030"}
Jul 11 03:43:04.879209 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:04.879178    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0339b6f-0473-42e8-854f-c943ab81ec25" containerID="1866c76b9d30609e62426505902b8240cbde547d6691f0dc05183f9d47ca98a5" exitCode=0
Jul 11 03:43:04.879309 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:04.879220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerDied","Data":"1866c76b9d30609e62426505902b8240cbde547d6691f0dc05183f9d47ca98a5"}
Jul 11 03:43:04.907941 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:04.907561    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt" podStartSLOduration=9.547240354 podStartE2EDuration="28.907538415s" totalImagesPullingTime="19.360298061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:04.905070607 +0000 UTC m=+29.774517150" watchObservedRunningTime="2026-07-11 03:43:04.907538415 +0000 UTC m=+29.776984977"
Jul 11 03:43:05.361011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.360912    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgr8c"]
Jul 11 03:43:05.361218 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.361125    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:05.361276 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:05.361233    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:43:05.363725 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.363701    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g28hp"]
Jul 11 03:43:05.363857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.363794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:05.363900 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:05.363879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:43:05.366659 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.366636    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlm47"]
Jul 11 03:43:05.367070 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.366745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:05.367070 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:05.366846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:43:05.882933 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.882901    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0339b6f-0473-42e8-854f-c943ab81ec25" containerID="5594a81317998e5543a79490cb40c9fe3716e085a49956002c0df81c6dc1f0e9" exitCode=0
Jul 11 03:43:05.883710 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:05.882976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerDied","Data":"5594a81317998e5543a79490cb40c9fe3716e085a49956002c0df81c6dc1f0e9"}
Jul 11 03:43:06.640182 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:06.640145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:06.640352 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:06.640268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:43:06.640352 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:06.640318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:06.640456 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:06.640433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:43:07.640016 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:07.639976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:07.640631 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:07.640141    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:43:08.640306 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:08.640278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:08.640306 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:08.640305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:08.640695 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:08.640391    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlm47" podUID="2be65b1e-0556-42b8-818d-e596d95d4a15"
Jul 11 03:43:08.640766 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:08.640726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g28hp" podUID="73a90a87-3cf0-4d8c-a59c-fbdfc116044e"
Jul 11 03:43:09.312485 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.312253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:09.312684 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.312453    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 11 03:43:09.312684 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.312578    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 11 03:43:09.312684 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.312592    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5hcxs for pod openshift-network-diagnostics/network-check-target-g28hp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:43:09.312684 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.312660    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs podName:73a90a87-3cf0-4d8c-a59c-fbdfc116044e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:41.312642334 +0000 UTC m=+66.182088892 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hcxs" (UniqueName: "kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs") pod "network-check-target-g28hp" (UID: "73a90a87-3cf0-4d8c-a59c-fbdfc116044e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 11 03:43:09.413394 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.413352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:09.413394 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.413402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:09.413626 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.413511    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 11 03:43:09.413626 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.413515    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:43:09.413626 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.413579    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs podName:2be65b1e-0556-42b8-818d-e596d95d4a15 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:41.413565362 +0000 UTC m=+66.283011922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs") pod "network-metrics-daemon-qlm47" (UID: "2be65b1e-0556-42b8-818d-e596d95d4a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 11 03:43:09.413626 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.413593    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret podName:5e41e790-86e5-4643-8afc-2f49245db72e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:41.413587414 +0000 UTC m=+66.283033955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret") pod "global-pull-secret-syncer-wgr8c" (UID: "5e41e790-86e5-4643-8afc-2f49245db72e") : object "kube-system"/"original-pull-secret" not registered
Jul 11 03:43:09.639919 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.639842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:09.640134 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:09.639985    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgr8c" podUID="5e41e790-86e5-4643-8afc-2f49245db72e"
Jul 11 03:43:09.946695 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.946586    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeReady"
Jul 11 03:43:09.947238 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.946735    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 11 03:43:09.979358 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.979325    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"]
Jul 11 03:43:09.996859 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.996829    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"]
Jul 11 03:43:09.997001 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.996960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.000459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.999574    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 11 03:43:10.000459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.999608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 11 03:43:10.000459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.999638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7kx2l\""
Jul 11 03:43:10.000459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:09.999682    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 11 03:43:10.000713 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.000601    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 11 03:43:10.012260 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.012234    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-585d7d9694-n76q9"]
Jul 11 03:43:10.012438 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.012374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.014877 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.014620    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-x6fmv\""
Jul 11 03:43:10.014877 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.014649    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 11 03:43:10.014877 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.014724    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 11 03:43:10.032466 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.032381    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"]
Jul 11 03:43:10.032681 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.032520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.035391 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.035130    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 11 03:43:10.035391 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.035232    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nr2fs\""
Jul 11 03:43:10.035391 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.035284    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 11 03:43:10.035721 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.035685    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 11 03:43:10.041668 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.041119    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 11 03:43:10.047412 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.047392    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxg8q"]
Jul 11 03:43:10.047525 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.047504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"
Jul 11 03:43:10.050032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.049837    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 11 03:43:10.050032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.049868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xk5qc\""
Jul 11 03:43:10.050032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.049843    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 11 03:43:10.065619 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.065588    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zzhr2"]
Jul 11 03:43:10.065735 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.065702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.069630 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.069606    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m57wb\""
Jul 11 03:43:10.069877 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.069861    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 11 03:43:10.070598 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.070578    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 11 03:43:10.071530 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.071242    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 11 03:43:10.083041 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083020    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"]
Jul 11 03:43:10.083166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"]
Jul 11 03:43:10.083166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"]
Jul 11 03:43:10.083166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxg8q"]
Jul 11 03:43:10.083166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585d7d9694-n76q9"]
Jul 11 03:43:10.083166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzhr2"]
Jul 11 03:43:10.083356 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.083177    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.087062 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.087043    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9qtgh\""
Jul 11 03:43:10.087704 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.087688    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 11 03:43:10.088198 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.088185    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 11 03:43:10.120898 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.120840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-bound-sa-token\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.120909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.120989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq8k\" (UniqueName: \"kubernetes.io/projected/92473190-1f60-4e48-9669-94df6487b33d-kube-api-access-qgq8k\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.121077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-image-registry-private-configuration\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-installation-pull-secrets\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121274 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.121274 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-kube-api-access-lsl88\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121274 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-trusted-ca\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121274 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92473190-1f60-4e48-9669-94df6487b33d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.121425 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92473190-1f60-4e48-9669-94df6487b33d-config\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.121425 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-certificates\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121504 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33e4097c-6799-48a6-9774-d01fbb0adc4e-ca-trust-extracted\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.121504 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.121462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/60537793-ea91-4fac-aa1b-a1026778eead-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.158913 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.158878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:10.221788 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/60537793-ea91-4fac-aa1b-a1026778eead-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.221788 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-bound-sa-token\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.221788 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxlq\" (UniqueName: \"kubernetes.io/projected/ee95f72b-4aa3-4eb2-9770-24cdd7109fdb-kube-api-access-khxlq\") pod \"network-check-source-869dcfcdcd-p8hsg\" (UID: \"ee95f72b-4aa3-4eb2-9770-24cdd7109fdb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq8k\" (UniqueName: \"kubernetes.io/projected/92473190-1f60-4e48-9669-94df6487b33d-kube-api-access-qgq8k\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb1cce9-f75d-42b2-b5ad-e123906216c0-tmp-dir\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.221949    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.221971    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585d7d9694-n76q9: secret "image-registry-tls" not found
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.221964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-image-registry-private-configuration\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.222020 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-installation-pull-secrets\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.222031    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls podName:33e4097c-6799-48a6-9774-d01fbb0adc4e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:10.722006988 +0000 UTC m=+35.591453536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls") pod "image-registry-585d7d9694-n76q9" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e") : secret "image-registry-tls" not found
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1cce9-f75d-42b2-b5ad-e123906216c0-config-volume\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-kube-api-access-lsl88\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-trusted-ca\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.222220    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92473190-1f60-4e48-9669-94df6487b33d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92473190-1f60-4e48-9669-94df6487b33d-config\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk4h\" (UniqueName: \"kubernetes.io/projected/14abdefe-12c5-4266-a648-06bd2fb711e1-kube-api-access-5tk4h\") pod \"ingress-canary-qxg8q\" (UID: \"14abdefe-12c5-4266-a648-06bd2fb711e1\") " pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.222380 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.222322    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert podName:60537793-ea91-4fac-aa1b-a1026778eead nodeName:}" failed. No retries permitted until 2026-07-11 03:43:10.722306036 +0000 UTC m=+35.591752581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-tsrsg" (UID: "60537793-ea91-4fac-aa1b-a1026778eead") : secret "networking-console-plugin-cert" not found
Jul 11 03:43:10.222853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.222853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-certificates\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.222853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14abdefe-12c5-4266-a648-06bd2fb711e1-cert\") pod \"ingress-canary-qxg8q\" (UID: \"14abdefe-12c5-4266-a648-06bd2fb711e1\") " pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.222853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w4q5\" (UniqueName: \"kubernetes.io/projected/1bb1cce9-f75d-42b2-b5ad-e123906216c0-kube-api-access-8w4q5\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.222853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33e4097c-6799-48a6-9774-d01fbb0adc4e-ca-trust-extracted\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.223051 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33e4097c-6799-48a6-9774-d01fbb0adc4e-ca-trust-extracted\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.223051 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-certificates\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.223051 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.222963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92473190-1f60-4e48-9669-94df6487b33d-config\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.223218 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.223050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/60537793-ea91-4fac-aa1b-a1026778eead-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.223266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.223231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-trusted-ca\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.226545 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.226502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92473190-1f60-4e48-9669-94df6487b33d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.226687 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.226620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-installation-pull-secrets\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.226687 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.226618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-image-registry-private-configuration\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.233515 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.233489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq8k\" (UniqueName: \"kubernetes.io/projected/92473190-1f60-4e48-9669-94df6487b33d-kube-api-access-qgq8k\") pod \"service-ca-operator-7695fc6c8c-fgh2n\" (UID: \"92473190-1f60-4e48-9669-94df6487b33d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.233877 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.233856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-bound-sa-token\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.233925 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.233912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-kube-api-access-lsl88\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.241746 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.241720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:10.311623 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.311595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"
Jul 11 03:43:10.322875 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.322841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk4h\" (UniqueName: \"kubernetes.io/projected/14abdefe-12c5-4266-a648-06bd2fb711e1-kube-api-access-5tk4h\") pod \"ingress-canary-qxg8q\" (UID: \"14abdefe-12c5-4266-a648-06bd2fb711e1\") " pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.323012 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.322881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.323012 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.322925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14abdefe-12c5-4266-a648-06bd2fb711e1-cert\") pod \"ingress-canary-qxg8q\" (UID: \"14abdefe-12c5-4266-a648-06bd2fb711e1\") " pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.323012 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.322947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w4q5\" (UniqueName: \"kubernetes.io/projected/1bb1cce9-f75d-42b2-b5ad-e123906216c0-kube-api-access-8w4q5\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.323012 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.322977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxlq\" (UniqueName: \"kubernetes.io/projected/ee95f72b-4aa3-4eb2-9770-24cdd7109fdb-kube-api-access-khxlq\") pod \"network-check-source-869dcfcdcd-p8hsg\" (UID: \"ee95f72b-4aa3-4eb2-9770-24cdd7109fdb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"
Jul 11 03:43:10.323244 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.323024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb1cce9-f75d-42b2-b5ad-e123906216c0-tmp-dir\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.323244 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.323076    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 11 03:43:10.323244 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.323138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1cce9-f75d-42b2-b5ad-e123906216c0-config-volume\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.323244 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.323174    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls podName:1bb1cce9-f75d-42b2-b5ad-e123906216c0 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:10.823153972 +0000 UTC m=+35.692600514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls") pod "dns-default-zzhr2" (UID: "1bb1cce9-f75d-42b2-b5ad-e123906216c0") : secret "dns-default-metrics-tls" not found
Jul 11 03:43:10.323455 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.323407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb1cce9-f75d-42b2-b5ad-e123906216c0-tmp-dir\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.323655 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.323639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1cce9-f75d-42b2-b5ad-e123906216c0-config-volume\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.325521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.325502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14abdefe-12c5-4266-a648-06bd2fb711e1-cert\") pod \"ingress-canary-qxg8q\" (UID: \"14abdefe-12c5-4266-a648-06bd2fb711e1\") " pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.334457 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.334428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w4q5\" (UniqueName: \"kubernetes.io/projected/1bb1cce9-f75d-42b2-b5ad-e123906216c0-kube-api-access-8w4q5\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.334710 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.334692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk4h\" (UniqueName: \"kubernetes.io/projected/14abdefe-12c5-4266-a648-06bd2fb711e1-kube-api-access-5tk4h\") pod \"ingress-canary-qxg8q\" (UID: \"14abdefe-12c5-4266-a648-06bd2fb711e1\") " pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.334965 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.334946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxlq\" (UniqueName: \"kubernetes.io/projected/ee95f72b-4aa3-4eb2-9770-24cdd7109fdb-kube-api-access-khxlq\") pod \"network-check-source-869dcfcdcd-p8hsg\" (UID: \"ee95f72b-4aa3-4eb2-9770-24cdd7109fdb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"
Jul 11 03:43:10.357040 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.357005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"
Jul 11 03:43:10.380885 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.380855    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxg8q"
Jul 11 03:43:10.491913 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.491858    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n"]
Jul 11 03:43:10.509760 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.509730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg"]
Jul 11 03:43:10.514720 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:10.514693    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92473190_1f60_4e48_9669_94df6487b33d.slice/crio-a0c32249bfed7848187850a9278a2777bc9af2d51bfe95b76aebc551cb572fcd WatchSource:0}: Error finding container a0c32249bfed7848187850a9278a2777bc9af2d51bfe95b76aebc551cb572fcd: Status 404 returned error can't find the container with id a0c32249bfed7848187850a9278a2777bc9af2d51bfe95b76aebc551cb572fcd
Jul 11 03:43:10.515123 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:10.515080    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee95f72b_4aa3_4eb2_9770_24cdd7109fdb.slice/crio-d1c4aefd0b4903a14f60a218487daa6f8b6584df592746196433a72d8a7eb174 WatchSource:0}: Error finding container d1c4aefd0b4903a14f60a218487daa6f8b6584df592746196433a72d8a7eb174: Status 404 returned error can't find the container with id d1c4aefd0b4903a14f60a218487daa6f8b6584df592746196433a72d8a7eb174
Jul 11 03:43:10.541248 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.541221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxg8q"]
Jul 11 03:43:10.544513 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:10.544484    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14abdefe_12c5_4266_a648_06bd2fb711e1.slice/crio-b8666a81714a695109a07463fba05e7f1b2826ba3397703070e4b9518c6c820b WatchSource:0}: Error finding container b8666a81714a695109a07463fba05e7f1b2826ba3397703070e4b9518c6c820b: Status 404 returned error can't find the container with id b8666a81714a695109a07463fba05e7f1b2826ba3397703070e4b9518c6c820b
Jul 11 03:43:10.639668 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.639511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:10.639794 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.639565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:10.642011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.641987    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 11 03:43:10.642130 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.642011    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-57qws\""
Jul 11 03:43:10.642130 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.642020    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6d8m4\""
Jul 11 03:43:10.723708 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.723672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:10.727729 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.727704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:10.728041 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.727760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:10.728041 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.727894    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 11 03:43:10.728041 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.727918    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 11 03:43:10.728041 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.727919    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585d7d9694-n76q9: secret "image-registry-tls" not found
Jul 11 03:43:10.728041 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.727978    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert podName:60537793-ea91-4fac-aa1b-a1026778eead nodeName:}" failed. No retries permitted until 2026-07-11 03:43:11.72795833 +0000 UTC m=+36.597404872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-tsrsg" (UID: "60537793-ea91-4fac-aa1b-a1026778eead") : secret "networking-console-plugin-cert" not found
Jul 11 03:43:10.728041 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.727993    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls podName:33e4097c-6799-48a6-9774-d01fbb0adc4e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:11.727985634 +0000 UTC m=+36.597432175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls") pod "image-registry-585d7d9694-n76q9" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e") : secret "image-registry-tls" not found
Jul 11 03:43:10.828403 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.828368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:10.828563 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.828510    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 11 03:43:10.828612 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:10.828571    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls podName:1bb1cce9-f75d-42b2-b5ad-e123906216c0 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:11.828557257 +0000 UTC m=+36.698003802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls") pod "dns-default-zzhr2" (UID: "1bb1cce9-f75d-42b2-b5ad-e123906216c0") : secret "dns-default-metrics-tls" not found
Jul 11 03:43:10.897634 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.897604    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0339b6f-0473-42e8-854f-c943ab81ec25" containerID="306c808f3732d22ae9cb631d2e22edfe96e247b2612bcef3afb50a006e64d3cc" exitCode=0
Jul 11 03:43:10.897810 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.897679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerDied","Data":"306c808f3732d22ae9cb631d2e22edfe96e247b2612bcef3afb50a006e64d3cc"}
Jul 11 03:43:10.898970 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.898766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxg8q" event={"ID":"14abdefe-12c5-4266-a648-06bd2fb711e1","Type":"ContainerStarted","Data":"b8666a81714a695109a07463fba05e7f1b2826ba3397703070e4b9518c6c820b"}
Jul 11 03:43:10.899877 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.899848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg" event={"ID":"ee95f72b-4aa3-4eb2-9770-24cdd7109fdb","Type":"ContainerStarted","Data":"d1c4aefd0b4903a14f60a218487daa6f8b6584df592746196433a72d8a7eb174"}
Jul 11 03:43:10.900872 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:10.900833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n" event={"ID":"92473190-1f60-4e48-9669-94df6487b33d","Type":"ContainerStarted","Data":"a0c32249bfed7848187850a9278a2777bc9af2d51bfe95b76aebc551cb572fcd"}
Jul 11 03:43:11.641533 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.641502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:11.643791 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.643768    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.739471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.739613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.739770    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.739786    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585d7d9694-n76q9: secret "image-registry-tls" not found
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.739850    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls podName:33e4097c-6799-48a6-9774-d01fbb0adc4e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:13.739828815 +0000 UTC m=+38.609275363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls") pod "image-registry-585d7d9694-n76q9" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e") : secret "image-registry-tls" not found
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.740282    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 11 03:43:11.740363 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.740327    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert podName:60537793-ea91-4fac-aa1b-a1026778eead nodeName:}" failed. No retries permitted until 2026-07-11 03:43:13.740312192 +0000 UTC m=+38.609758735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-tsrsg" (UID: "60537793-ea91-4fac-aa1b-a1026778eead") : secret "networking-console-plugin-cert" not found
Jul 11 03:43:11.840703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.840663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:11.840889 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.840838    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 11 03:43:11.840931 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:11.840902    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls podName:1bb1cce9-f75d-42b2-b5ad-e123906216c0 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:13.840883216 +0000 UTC m=+38.710329761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls") pod "dns-default-zzhr2" (UID: "1bb1cce9-f75d-42b2-b5ad-e123906216c0") : secret "dns-default-metrics-tls" not found
Jul 11 03:43:11.908866 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.908779    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0339b6f-0473-42e8-854f-c943ab81ec25" containerID="53bbb07696c0682382be868d1c48a725ee276076e84dd6f1f861ef796dbd7086" exitCode=0
Jul 11 03:43:11.908866 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:11.908853    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerDied","Data":"53bbb07696c0682382be868d1c48a725ee276076e84dd6f1f861ef796dbd7086"}
Jul 11 03:43:12.137638 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:12.137600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:12.237722 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:12.237693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:12.918887 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:12.918856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" event={"ID":"e0339b6f-0473-42e8-854f-c943ab81ec25","Type":"ContainerStarted","Data":"49cdbdcba1cf824a299b30b5d790a2c13bbf4580c2998d00ebe089002df5aeb7"}
Jul 11 03:43:12.940720 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:12.940658    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vlw6w" podStartSLOduration=7.152960715 podStartE2EDuration="36.940638391s" totalImagesPullingTime="29.787677676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:12.94050774 +0000 UTC m=+37.809954341" watchObservedRunningTime="2026-07-11 03:43:12.940638391 +0000 UTC m=+37.810084958"
Jul 11 03:43:13.067419 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:13.067382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:13.119608 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:13.119573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:13.757698 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:13.757660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:13.757868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:13.757731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:13.757868 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.757820    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 11 03:43:13.757868 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.757834    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585d7d9694-n76q9: secret "image-registry-tls" not found
Jul 11 03:43:13.757969 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.757877    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 11 03:43:13.757969 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.757891    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls podName:33e4097c-6799-48a6-9774-d01fbb0adc4e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:17.757875484 +0000 UTC m=+42.627322045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls") pod "image-registry-585d7d9694-n76q9" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e") : secret "image-registry-tls" not found
Jul 11 03:43:13.757969 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.757945    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert podName:60537793-ea91-4fac-aa1b-a1026778eead nodeName:}" failed. No retries permitted until 2026-07-11 03:43:17.757927985 +0000 UTC m=+42.627374527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-tsrsg" (UID: "60537793-ea91-4fac-aa1b-a1026778eead") : secret "networking-console-plugin-cert" not found
Jul 11 03:43:13.858690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:13.858648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:13.858850 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.858806    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 11 03:43:13.858889 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:13.858870    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls podName:1bb1cce9-f75d-42b2-b5ad-e123906216c0 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:17.858855237 +0000 UTC m=+42.728301779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls") pod "dns-default-zzhr2" (UID: "1bb1cce9-f75d-42b2-b5ad-e123906216c0") : secret "dns-default-metrics-tls" not found
Jul 11 03:43:14.097919 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:14.097802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:14.149687 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:14.149653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:15.929174 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:15.928914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxg8q" event={"ID":"14abdefe-12c5-4266-a648-06bd2fb711e1","Type":"ContainerStarted","Data":"35889523c4e07aeccab19fb04e2a4138b876104b33fbc3e68705861c99ecae99"}
Jul 11 03:43:15.930465 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:15.930435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg" event={"ID":"ee95f72b-4aa3-4eb2-9770-24cdd7109fdb","Type":"ContainerStarted","Data":"24114893d78a1de57f7859b948eb3839653eef5a6c4ddafe458f2ffbf3a01058"}
Jul 11 03:43:15.931967 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:15.931944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n" event={"ID":"92473190-1f60-4e48-9669-94df6487b33d","Type":"ContainerStarted","Data":"3dceb6b1940926c9e4d8b4c6a6152d24d2ea56dd814fd1fcb73a7d70e584688e"}
Jul 11 03:43:15.946063 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:15.946014    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxg8q" podStartSLOduration=2.271850372 podStartE2EDuration="6.945998077s" totalImagesPullingTime="4.674147705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:15.945721353 +0000 UTC m=+40.815167930" watchObservedRunningTime="2026-07-11 03:43:15.945998077 +0000 UTC m=+40.815444637"
Jul 11 03:43:15.960740 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:15.960690    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8hsg" podStartSLOduration=34.251092439 podStartE2EDuration="38.960670855s" totalImagesPullingTime="4.709578416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:15.959416807 +0000 UTC m=+40.828863371" watchObservedRunningTime="2026-07-11 03:43:15.960670855 +0000 UTC m=+40.830117400"
Jul 11 03:43:15.974473 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:15.974426    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n" podStartSLOduration=35.27503903 podStartE2EDuration="39.974409224s" totalImagesPullingTime="4.699370194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:15.97381366 +0000 UTC m=+40.843260226" watchObservedRunningTime="2026-07-11 03:43:15.974409224 +0000 UTC m=+40.843855789"
Jul 11 03:43:16.724002 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:16.723962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:16.780051 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:16.780020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.012241 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.012206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.162335 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.162294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.213684 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.213651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.364060 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.363987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.464178 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.464140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.518230 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.518195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.764259 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.764230    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.792442 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.792406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:17.792596 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.792520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:17.792596 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.792551    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 11 03:43:17.792707 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.792613    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert podName:60537793-ea91-4fac-aa1b-a1026778eead nodeName:}" failed. No retries permitted until 2026-07-11 03:43:25.792597927 +0000 UTC m=+50.662044473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-tsrsg" (UID: "60537793-ea91-4fac-aa1b-a1026778eead") : secret "networking-console-plugin-cert" not found
Jul 11 03:43:17.792707 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.792633    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 11 03:43:17.792707 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.792647    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585d7d9694-n76q9: secret "image-registry-tls" not found
Jul 11 03:43:17.792821 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.792713    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls podName:33e4097c-6799-48a6-9774-d01fbb0adc4e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:25.792696201 +0000 UTC m=+50.662142762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls") pod "image-registry-585d7d9694-n76q9" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e") : secret "image-registry-tls" not found
Jul 11 03:43:17.833037 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.833006    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:17.893163 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:17.893138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:17.893273 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.893250    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 11 03:43:17.893315 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:17.893300    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls podName:1bb1cce9-f75d-42b2-b5ad-e123906216c0 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:25.893287673 +0000 UTC m=+50.762734215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls") pod "dns-default-zzhr2" (UID: "1bb1cce9-f75d-42b2-b5ad-e123906216c0") : secret "dns-default-metrics-tls" not found
Jul 11 03:43:18.926428 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:18.926397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:19.074781 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:19.074747    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:19.874081 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:19.874047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:21.965093 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:21.965063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:22.027081 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:22.027051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:22.364449 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:22.364414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:23.722550 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:23.722511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:23.892188 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:23.892159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:24.000860 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:24.000831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:24.099083 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:24.099050    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:24.199139 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:24.199083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:24.299808 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:24.299735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:24.624155 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:24.624068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:25.423240 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:25.423207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:25.818403 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:25.818368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:25.857652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:25.857620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:25.857814 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:25.857684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:25.857814 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.857778    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 11 03:43:25.857883 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.857815    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585d7d9694-n76q9: secret "image-registry-tls" not found
Jul 11 03:43:25.857883 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.857835    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 11 03:43:25.857945 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.857884    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls podName:33e4097c-6799-48a6-9774-d01fbb0adc4e nodeName:}" failed. No retries permitted until 2026-07-11 03:43:41.857866875 +0000 UTC m=+66.727313437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls") pod "image-registry-585d7d9694-n76q9" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e") : secret "image-registry-tls" not found
Jul 11 03:43:25.857945 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.857898    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert podName:60537793-ea91-4fac-aa1b-a1026778eead nodeName:}" failed. No retries permitted until 2026-07-11 03:43:41.857892877 +0000 UTC m=+66.727339418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-tsrsg" (UID: "60537793-ea91-4fac-aa1b-a1026778eead") : secret "networking-console-plugin-cert" not found
Jul 11 03:43:25.958334 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:25.958309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:25.958495 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.958449    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 11 03:43:25.958539 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:25.958504    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls podName:1bb1cce9-f75d-42b2-b5ad-e123906216c0 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:41.95848818 +0000 UTC m=+66.827934722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls") pod "dns-default-zzhr2" (UID: "1bb1cce9-f75d-42b2-b5ad-e123906216c0") : secret "dns-default-metrics-tls" not found
Jul 11 03:43:26.029585 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:26.029553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:26.225279 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:26.225199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:26.425450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:26.425415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:26.723083 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:26.723051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:27.345366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:27.345326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:27.446152 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:27.446120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:27.618415 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:27.618339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:27.819974 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:27.819937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:28.020888 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:28.020852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:28.219092 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:28.219061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:28.269779 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:28.269751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:28.420917 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:28.420835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:29.168912 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:29.168883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:29.269802 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:29.269768    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:29.420010 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:29.419931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:29.617145 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:29.617113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:30.021926 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:30.021895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:30.123111 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:30.123059    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:30.227472 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:30.227445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:30.615993 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:30.615963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:31.223628 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:31.223596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:31.619774 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:31.619744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:32.019347 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:32.019319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:32.421669 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:32.421569    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:32.617801 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:32.617754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:32.758743 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:32.758712    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:32.857859 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:32.857833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:32.909819 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:32.909783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:33.021823 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:33.021744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:33.617366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:33.617331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:33.822428 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:33.822397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:34.021482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:34.021433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:35.894303 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:35.894275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8r9wt"
Jul 11 03:43:40.506889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.506843    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"]
Jul 11 03:43:40.511245 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.511227    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"]
Jul 11 03:43:40.511359 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.511342    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.514287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.514267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"
Jul 11 03:43:40.518811 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.518796    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 11 03:43:40.519351 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.519336    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 11 03:43:40.519419 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.519353    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wf54p\""
Jul 11 03:43:40.519925 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.519909    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 11 03:43:40.521424 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.521406    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 11 03:43:40.523532 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.523516    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-65wnw\""
Jul 11 03:43:40.526634 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.526618    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 11 03:43:40.540902 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.540881    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"]
Jul 11 03:43:40.541791 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.541770    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"]
Jul 11 03:43:40.570669 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.570646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz48v\" (UniqueName: \"kubernetes.io/projected/5f7f2647-a27a-48ab-a71f-e78bd8d6581b-kube-api-access-kz48v\") pod \"cluster-samples-operator-65db858894-9pgrh\" (UID: \"5f7f2647-a27a-48ab-a71f-e78bd8d6581b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.570774 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.570691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f7f2647-a27a-48ab-a71f-e78bd8d6581b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9pgrh\" (UID: \"5f7f2647-a27a-48ab-a71f-e78bd8d6581b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.570774 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.570766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9268b\" (UniqueName: \"kubernetes.io/projected/b316cf50-eb33-4e1c-85a1-d800cb5ce977-kube-api-access-9268b\") pod \"volume-data-source-validator-78495597f6-vz9z4\" (UID: \"b316cf50-eb33-4e1c-85a1-d800cb5ce977\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"
Jul 11 03:43:40.612870 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.612835    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"]
Jul 11 03:43:40.616033 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.616014    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.620842 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.620820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 11 03:43:40.621088 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.621073    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 11 03:43:40.621690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.621669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 11 03:43:40.621793 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.621669    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 11 03:43:40.621949 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.621929    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 11 03:43:40.622030 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.621985    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 11 03:43:40.622391 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.622275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 11 03:43:40.631765 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.631740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"]
Jul 11 03:43:40.671196 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9268b\" (UniqueName: \"kubernetes.io/projected/b316cf50-eb33-4e1c-85a1-d800cb5ce977-kube-api-access-9268b\") pod \"volume-data-source-validator-78495597f6-vz9z4\" (UID: \"b316cf50-eb33-4e1c-85a1-d800cb5ce977\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"
Jul 11 03:43:40.671196 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f7f2647-a27a-48ab-a71f-e78bd8d6581b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9pgrh\" (UID: \"5f7f2647-a27a-48ab-a71f-e78bd8d6581b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.671482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.671482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.671482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-ca\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.671482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.671482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz48v\" (UniqueName: \"kubernetes.io/projected/5f7f2647-a27a-48ab-a71f-e78bd8d6581b-kube-api-access-kz48v\") pod \"cluster-samples-operator-65db858894-9pgrh\" (UID: \"5f7f2647-a27a-48ab-a71f-e78bd8d6581b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.671482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-hub\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.671716 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.671577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8q9c\" (UniqueName: \"kubernetes.io/projected/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-kube-api-access-n8q9c\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.675729 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.675702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f7f2647-a27a-48ab-a71f-e78bd8d6581b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9pgrh\" (UID: \"5f7f2647-a27a-48ab-a71f-e78bd8d6581b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.678025 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.677992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:40.682810 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.682788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9268b\" (UniqueName: \"kubernetes.io/projected/b316cf50-eb33-4e1c-85a1-d800cb5ce977-kube-api-access-9268b\") pod \"volume-data-source-validator-78495597f6-vz9z4\" (UID: \"b316cf50-eb33-4e1c-85a1-d800cb5ce977\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"
Jul 11 03:43:40.683034 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.683020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz48v\" (UniqueName: \"kubernetes.io/projected/5f7f2647-a27a-48ab-a71f-e78bd8d6581b-kube-api-access-kz48v\") pod \"cluster-samples-operator-65db858894-9pgrh\" (UID: \"5f7f2647-a27a-48ab-a71f-e78bd8d6581b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.753487 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.753450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:40.772981 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.772894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.772981 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.772960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.773227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.772999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-ca\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.773227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.773033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.773227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.773058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-hub\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.773227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.773083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8q9c\" (UniqueName: \"kubernetes.io/projected/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-kube-api-access-n8q9c\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.773888 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.773866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.775482 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.775458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-ca\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.775700 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.775683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.776028 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.776004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-hub\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.776131 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.776067    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.786168 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.786145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8q9c\" (UniqueName: \"kubernetes.io/projected/8b98e363-f8a4-4ffd-8f36-9b314a5d0a19-kube-api-access-n8q9c\") pod \"cluster-proxy-proxy-agent-6b65599c4d-sfrnz\" (UID: \"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.821003 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.820977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"
Jul 11 03:43:40.826194 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.826167    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"
Jul 11 03:43:40.853365 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.853340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:40.937749 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.937564    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"
Jul 11 03:43:40.955234 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.955208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:40.978382 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.978279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh"]
Jul 11 03:43:40.998198 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:40.998060    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4"]
Jul 11 03:43:41.011752 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:41.011717    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb316cf50_eb33_4e1c_85a1_d800cb5ce977.slice/crio-2435c8e90327b5d5d7cc556810de03fb6a7fe72a3b9f4115b33136430c1c0416 WatchSource:0}: Error finding container 2435c8e90327b5d5d7cc556810de03fb6a7fe72a3b9f4115b33136430c1c0416: Status 404 returned error can't find the container with id 2435c8e90327b5d5d7cc556810de03fb6a7fe72a3b9f4115b33136430c1c0416
Jul 11 03:43:41.052362 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.052337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:41.090288 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.090261    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz"]
Jul 11 03:43:41.092909 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:41.092879    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b98e363_f8a4_4ffd_8f36_9b314a5d0a19.slice/crio-b7847edb1bc665b5e450f1ab478a9d8a460d61e9878e5b0d5d1b896986653b57 WatchSource:0}: Error finding container b7847edb1bc665b5e450f1ab478a9d8a460d61e9878e5b0d5d1b896986653b57: Status 404 returned error can't find the container with id b7847edb1bc665b5e450f1ab478a9d8a460d61e9878e5b0d5d1b896986653b57
Jul 11 03:43:41.379907 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.379814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:41.382489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.382468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcxs\" (UniqueName: \"kubernetes.io/projected/73a90a87-3cf0-4d8c-a59c-fbdfc116044e-kube-api-access-5hcxs\") pod \"network-check-target-g28hp\" (UID: \"73a90a87-3cf0-4d8c-a59c-fbdfc116044e\") " pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:41.480560 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.480530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:41.480560 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.480575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:41.482891 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.482873    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 11 03:43:41.482946 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.482881    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 11 03:43:41.493624 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.493592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e41e790-86e5-4643-8afc-2f49245db72e-original-pull-secret\") pod \"global-pull-secret-syncer-wgr8c\" (UID: \"5e41e790-86e5-4643-8afc-2f49245db72e\") " pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:41.493745 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.493607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be65b1e-0556-42b8-818d-e596d95d4a15-metrics-certs\") pod \"network-metrics-daemon-qlm47\" (UID: \"2be65b1e-0556-42b8-818d-e596d95d4a15\") " pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:41.551184 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.551152    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6d8m4\""
Jul 11 03:43:41.557036 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.557000    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-57qws\""
Jul 11 03:43:41.559778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.559757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlm47"
Jul 11 03:43:41.565940 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.565922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:41.653294 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.653204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgr8c"
Jul 11 03:43:41.736689 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.736580    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlm47"]
Jul 11 03:43:41.739681 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:41.739640    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be65b1e_0556_42b8_818d_e596d95d4a15.slice/crio-3fe261b9b2041669455e0728a27476c7dd328457626b80f1709b48383f153a47 WatchSource:0}: Error finding container 3fe261b9b2041669455e0728a27476c7dd328457626b80f1709b48383f153a47: Status 404 returned error can't find the container with id 3fe261b9b2041669455e0728a27476c7dd328457626b80f1709b48383f153a47
Jul 11 03:43:41.752891 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.752528    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g28hp"]
Jul 11 03:43:41.762312 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:41.762276    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a90a87_3cf0_4d8c_a59c_fbdfc116044e.slice/crio-6c5579754b992b2340b4a89bff5bec33494090cfc240aa653bbd8b42f3eced1a WatchSource:0}: Error finding container 6c5579754b992b2340b4a89bff5bec33494090cfc240aa653bbd8b42f3eced1a: Status 404 returned error can't find the container with id 6c5579754b992b2340b4a89bff5bec33494090cfc240aa653bbd8b42f3eced1a
Jul 11 03:43:41.763687 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.763598    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:41.809060 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.809030    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgr8c"]
Jul 11 03:43:41.815272 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:41.815239    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e41e790_86e5_4643_8afc_2f49245db72e.slice/crio-86c39bd8f87a237c8c8df4072d15783539e2005d2de006cf2e9d8da318837de0 WatchSource:0}: Error finding container 86c39bd8f87a237c8c8df4072d15783539e2005d2de006cf2e9d8da318837de0: Status 404 returned error can't find the container with id 86c39bd8f87a237c8c8df4072d15783539e2005d2de006cf2e9d8da318837de0
Jul 11 03:43:41.833204 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.833124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:41.884539 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.884511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:41.884668 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.884566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:41.887816 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.887748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/60537793-ea91-4fac-aa1b-a1026778eead-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tsrsg\" (UID: \"60537793-ea91-4fac-aa1b-a1026778eead\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:41.888561 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.888544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"image-registry-585d7d9694-n76q9\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") " pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:41.985503 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.985464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:41.988240 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:41.988211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bb1cce9-f75d-42b2-b5ad-e123906216c0-metrics-tls\") pod \"dns-default-zzhr2\" (UID: \"1bb1cce9-f75d-42b2-b5ad-e123906216c0\") " pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:42.000048 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.000014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g28hp" event={"ID":"73a90a87-3cf0-4d8c-a59c-fbdfc116044e","Type":"ContainerStarted","Data":"39f70937daab8dea4f43a69b006d65588e5d26a28ab83597cb5a2efb4733fa7c"}
Jul 11 03:43:42.000213 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.000078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g28hp" event={"ID":"73a90a87-3cf0-4d8c-a59c-fbdfc116044e","Type":"ContainerStarted","Data":"6c5579754b992b2340b4a89bff5bec33494090cfc240aa653bbd8b42f3eced1a"}
Jul 11 03:43:42.000292 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.000248    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:43:42.001758 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.001723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz" event={"ID":"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19","Type":"ContainerStarted","Data":"b7847edb1bc665b5e450f1ab478a9d8a460d61e9878e5b0d5d1b896986653b57"}
Jul 11 03:43:42.004368 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.004311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4" event={"ID":"b316cf50-eb33-4e1c-85a1-d800cb5ce977","Type":"ContainerStarted","Data":"2435c8e90327b5d5d7cc556810de03fb6a7fe72a3b9f4115b33136430c1c0416"}
Jul 11 03:43:42.005618 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.005594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlm47" event={"ID":"2be65b1e-0556-42b8-818d-e596d95d4a15","Type":"ContainerStarted","Data":"3fe261b9b2041669455e0728a27476c7dd328457626b80f1709b48383f153a47"}
Jul 11 03:43:42.007194 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.007169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgr8c" event={"ID":"5e41e790-86e5-4643-8afc-2f49245db72e","Type":"ContainerStarted","Data":"86c39bd8f87a237c8c8df4072d15783539e2005d2de006cf2e9d8da318837de0"}
Jul 11 03:43:42.008372 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.008353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh" event={"ID":"5f7f2647-a27a-48ab-a71f-e78bd8d6581b","Type":"ContainerStarted","Data":"404e9bac654d287b4d084504f19eb61e79db2793e7ed7f7bc6a06821c788e937"}
Jul 11 03:43:42.015480 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.015434    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g28hp" podStartSLOduration=66.015420869 podStartE2EDuration="1m6.015420869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:42.01457235 +0000 UTC m=+66.884018926" watchObservedRunningTime="2026-07-11 03:43:42.015420869 +0000 UTC m=+66.884867432"
Jul 11 03:43:42.126768 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.126549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-x6fmv\""
Jul 11 03:43:42.135127 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.134957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"
Jul 11 03:43:42.147445 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.147089    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nr2fs\""
Jul 11 03:43:42.155575 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.155426    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:42.195812 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.195599    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9qtgh\""
Jul 11 03:43:42.204375 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.204162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:42.755851 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.755746    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585d7d9694-n76q9"]
Jul 11 03:43:42.761824 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:42.761756    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e4097c_6799_48a6_9774_d01fbb0adc4e.slice/crio-bcd2becef3b984b153c2ea57dc23e9bb19b33a3744d127e1b1d90aec57feafd1 WatchSource:0}: Error finding container bcd2becef3b984b153c2ea57dc23e9bb19b33a3744d127e1b1d90aec57feafd1: Status 404 returned error can't find the container with id bcd2becef3b984b153c2ea57dc23e9bb19b33a3744d127e1b1d90aec57feafd1
Jul 11 03:43:42.762769 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.762743    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzhr2"]
Jul 11 03:43:42.767955 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:42.767919    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb1cce9_f75d_42b2_b5ad_e123906216c0.slice/crio-2b057fe6a6aa5e2f4ce0bcd5de40b01c13e96de97e9d4ca0518e83276da62f8e WatchSource:0}: Error finding container 2b057fe6a6aa5e2f4ce0bcd5de40b01c13e96de97e9d4ca0518e83276da62f8e: Status 404 returned error can't find the container with id 2b057fe6a6aa5e2f4ce0bcd5de40b01c13e96de97e9d4ca0518e83276da62f8e
Jul 11 03:43:42.782031 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.781946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tsrsg"]
Jul 11 03:43:42.819159 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.818983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:42.875178 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:42.875067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:43.031424 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:43.031371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585d7d9694-n76q9" event={"ID":"33e4097c-6799-48a6-9774-d01fbb0adc4e","Type":"ContainerStarted","Data":"bcd2becef3b984b153c2ea57dc23e9bb19b33a3744d127e1b1d90aec57feafd1"}
Jul 11 03:43:43.034343 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:43.034287    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4" event={"ID":"b316cf50-eb33-4e1c-85a1-d800cb5ce977","Type":"ContainerStarted","Data":"6db4cbd93dfa8b4b11dc3c5b4def9eefb5a9de1e846bfb96c98a857e5b9f1dbe"}
Jul 11 03:43:43.037790 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:43.037744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzhr2" event={"ID":"1bb1cce9-f75d-42b2-b5ad-e123906216c0","Type":"ContainerStarted","Data":"2b057fe6a6aa5e2f4ce0bcd5de40b01c13e96de97e9d4ca0518e83276da62f8e"}
Jul 11 03:43:43.051963 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:43.051867    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vz9z4" podStartSLOduration=1.509727866 podStartE2EDuration="3.051846472s" totalImagesPullingTime="1.542118606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:43.050275944 +0000 UTC m=+67.919722502" watchObservedRunningTime="2026-07-11 03:43:43.051846472 +0000 UTC m=+67.921293037"
Jul 11 03:43:43.129389 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:43.127133    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60537793_ea91_4fac_aa1b_a1026778eead.slice/crio-68d5e416fc850f0ff597570ff46a824ce0dfb177bb0c2f748ee72c09f4ae25e3 WatchSource:0}: Error finding container 68d5e416fc850f0ff597570ff46a824ce0dfb177bb0c2f748ee72c09f4ae25e3: Status 404 returned error can't find the container with id 68d5e416fc850f0ff597570ff46a824ce0dfb177bb0c2f748ee72c09f4ae25e3
Jul 11 03:43:43.129389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:43.127917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:43.229469 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:43.229424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:44.043430 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:44.043378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg" event={"ID":"60537793-ea91-4fac-aa1b-a1026778eead","Type":"ContainerStarted","Data":"68d5e416fc850f0ff597570ff46a824ce0dfb177bb0c2f748ee72c09f4ae25e3"}
Jul 11 03:43:45.050363 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:45.050283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585d7d9694-n76q9" event={"ID":"33e4097c-6799-48a6-9774-d01fbb0adc4e","Type":"ContainerStarted","Data":"14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040"}
Jul 11 03:43:45.051895 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:45.051071    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:43:45.073133 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:45.072153    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-585d7d9694-n76q9" podStartSLOduration=69.072131555 podStartE2EDuration="1m9.072131555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:45.06947204 +0000 UTC m=+69.938918605" watchObservedRunningTime="2026-07-11 03:43:45.072131555 +0000 UTC m=+69.941578120"
Jul 11 03:43:48.532268 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:48.532237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:48.631127 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:48.631071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:48.728402 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:48.728362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:48.818804 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:48.818722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:48.886301 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:48.886273    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:48.954014 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:48.953988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.065528 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.065503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlm47" event={"ID":"2be65b1e-0556-42b8-818d-e596d95d4a15","Type":"ContainerStarted","Data":"139afbf33a57e6b5f5d97053413774222e0b499c2891c9a7f7d2bfd78c7e5524"}
Jul 11 03:43:49.067132 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.067069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgr8c" event={"ID":"5e41e790-86e5-4643-8afc-2f49245db72e","Type":"ContainerStarted","Data":"a6e6b2302b92d60ddc9cacfbf126e2b15c72f1c74fa2c418dd85988b0d52507c"}
Jul 11 03:43:49.068418 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.068401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh" event={"ID":"5f7f2647-a27a-48ab-a71f-e78bd8d6581b","Type":"ContainerStarted","Data":"34a60ffb46746e0f68965a2ff42c6c9856cc24b4aa84bf50bea49759d45783c1"}
Jul 11 03:43:49.069998 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.069914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz" event={"ID":"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19","Type":"ContainerStarted","Data":"ea3dd84fc558168cc487df75e2f540c2e81d07b6277aebaa55910235937c8dde"}
Jul 11 03:43:49.130306 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.127948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.320881 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.320769    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.353840 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.353807    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-crc28"]
Jul 11 03:43:49.357022 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.356999    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.359385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.359361    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ddl6z\""
Jul 11 03:43:49.360158 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.360136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 11 03:43:49.360295 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.360262    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 11 03:43:49.360295 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.360274    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 11 03:43:49.360432 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.360267    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 11 03:43:49.374864 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.374842    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-crc28"]
Jul 11 03:43:49.403623 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.403589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.448225 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.448194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.448416 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.448252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qck7l\" (UniqueName: \"kubernetes.io/projected/c26922ba-a360-408d-a2ac-e150405b17b8-kube-api-access-qck7l\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.448416 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.448287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c26922ba-a360-408d-a2ac-e150405b17b8-data-volume\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.448416 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.448362    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.448536 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.448452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c26922ba-a360-408d-a2ac-e150405b17b8-crio-socket\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.549692 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.549655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c26922ba-a360-408d-a2ac-e150405b17b8-data-volume\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.549692 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.549700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.549734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c26922ba-a360-408d-a2ac-e150405b17b8-crio-socket\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.549762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.549787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qck7l\" (UniqueName: \"kubernetes.io/projected/c26922ba-a360-408d-a2ac-e150405b17b8-kube-api-access-qck7l\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:49.549795    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:49.549865    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:50.049848645 +0000 UTC m=+74.919295187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : configmap "kube-rbac-proxy" not found
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.549883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c26922ba-a360-408d-a2ac-e150405b17b8-crio-socket\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:49.549941    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:49.550000    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:50.049981578 +0000 UTC m=+74.919428124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:49.550148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.550025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c26922ba-a360-408d-a2ac-e150405b17b8-data-volume\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.558376 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.558351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.560562 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.560543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qck7l\" (UniqueName: \"kubernetes.io/projected/c26922ba-a360-408d-a2ac-e150405b17b8-kube-api-access-qck7l\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:49.658967 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.658875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.767567 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.767528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:49.933933 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:49.933840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.036974 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.036938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.054149 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.054116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:50.054294 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.054163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:50.054294 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:50.054234    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 11 03:43:50.054294 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:50.054266    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:50.054294 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:50.054288    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:51.054273667 +0000 UTC m=+75.923720209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : configmap "kube-rbac-proxy" not found
Jul 11 03:43:50.054417 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:50.054319    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:51.054305542 +0000 UTC m=+75.923752084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:50.075565 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.075528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlm47" event={"ID":"2be65b1e-0556-42b8-818d-e596d95d4a15","Type":"ContainerStarted","Data":"afc17deedd6fe02156a34b505246d47f3eb763c0fe5dfa9f40a8527c3ebed9b1"}
Jul 11 03:43:50.077200 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.077178    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzhr2" event={"ID":"1bb1cce9-f75d-42b2-b5ad-e123906216c0","Type":"ContainerStarted","Data":"7ba7305f6ca1a634e052dc458a20c17a9cd25ea13c1d99fd7fe9b861aa7bd20d"}
Jul 11 03:43:50.077278 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.077228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzhr2" event={"ID":"1bb1cce9-f75d-42b2-b5ad-e123906216c0","Type":"ContainerStarted","Data":"2ac542246402f13e70e536f732adf6689234e9b67fd29e13a7c2b699f88948d3"}
Jul 11 03:43:50.077278 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.077261    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:43:50.078766 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.078744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg" event={"ID":"60537793-ea91-4fac-aa1b-a1026778eead","Type":"ContainerStarted","Data":"c308e1c87c82567a1a478b1b3d9d7c37a4e71ffa6515fe8081ab2996fbc26199"}
Jul 11 03:43:50.080571 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.080535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh" event={"ID":"5f7f2647-a27a-48ab-a71f-e78bd8d6581b","Type":"ContainerStarted","Data":"3f50c3048e3d7402a6509cac0775a001e9a5a5f0969627ae47ddf215bf27cbcc"}
Jul 11 03:43:50.082648 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.082626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz" event={"ID":"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19","Type":"ContainerStarted","Data":"bf6f4ab9a79e77c12bd680eec155d340f62ea79c67d146b1699498ebf57b7881"}
Jul 11 03:43:50.082735 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.082659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz" event={"ID":"8b98e363-f8a4-4ffd-8f36-9b314a5d0a19","Type":"ContainerStarted","Data":"2a23d47330b70e25a3a58d4a545090bde6225fee5b18dda420e36c5ba03b4553"}
Jul 11 03:43:50.092409 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.092364    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qlm47" podStartSLOduration=67.776427829 podStartE2EDuration="1m14.092352031s" totalImagesPullingTime="6.315924202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.091388763 +0000 UTC m=+74.960835328" watchObservedRunningTime="2026-07-11 03:43:50.092352031 +0000 UTC m=+74.961798595"
Jul 11 03:43:50.103056 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.103033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.107694 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.107654    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zzhr2" podStartSLOduration=35.022756693 podStartE2EDuration="41.107637479s" totalImagesPullingTime="6.084880786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.106553876 +0000 UTC m=+74.976000439" watchObservedRunningTime="2026-07-11 03:43:50.107637479 +0000 UTC m=+74.977084043"
Jul 11 03:43:50.121831 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.121782    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9pgrh" podStartSLOduration=3.123433665 podStartE2EDuration="10.121765632s" totalImagesPullingTime="6.998331967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.121272506 +0000 UTC m=+74.990719069" watchObservedRunningTime="2026-07-11 03:43:50.121765632 +0000 UTC m=+74.991212243"
Jul 11 03:43:50.142229 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.142177    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65599c4d-sfrnz" podStartSLOduration=2.41370246 podStartE2EDuration="10.142160572s" totalImagesPullingTime="7.728458112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.14107553 +0000 UTC m=+75.010522108" watchObservedRunningTime="2026-07-11 03:43:50.142160572 +0000 UTC m=+75.011607135"
Jul 11 03:43:50.156640 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.156595    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-tsrsg" podStartSLOduration=40.42788319 podStartE2EDuration="46.156578706s" totalImagesPullingTime="5.728695516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.156262926 +0000 UTC m=+75.025709490" watchObservedRunningTime="2026-07-11 03:43:50.156578706 +0000 UTC m=+75.026025269"
Jul 11 03:43:50.166894 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.166865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.175091 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.175047    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wgr8c" podStartSLOduration=67.169438142 podStartE2EDuration="1m14.175031957s" totalImagesPullingTime="7.005593815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.174406625 +0000 UTC m=+75.043853190" watchObservedRunningTime="2026-07-11 03:43:50.175031957 +0000 UTC m=+75.044478536"
Jul 11 03:43:50.265329 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.265298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.365703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.365669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.483295 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.483258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:50.868976 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:50.868947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:51.063688 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.063655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:51.063857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.063701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:51.063857 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:51.063790    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 11 03:43:51.063857 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:51.063803    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:51.063857 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:51.063841    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:53.0638276 +0000 UTC m=+77.933274142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : configmap "kube-rbac-proxy" not found
Jul 11 03:43:51.063992 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:51.063870    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:53.063855686 +0000 UTC m=+77.933302230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:51.127471 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.127373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:51.498841 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.498811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:51.556148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.556078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:51.599689 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.599666    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzhr2_1bb1cce9-f75d-42b2-b5ad-e123906216c0/dns/0.log"
Jul 11 03:43:51.604559 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.604537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:51.800219 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:51.800146    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzhr2_1bb1cce9-f75d-42b2-b5ad-e123906216c0/kube-rbac-proxy/0.log"
Jul 11 03:43:52.206588 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.206516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:52.308992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.308956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:52.400074 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.400050    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rdtx4_21e8f34f-1131-4a4e-9830-7eba4d3135e4/dns-node-resolver/0.log"
Jul 11 03:43:52.407748 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.407726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:52.602235 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.602212    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-585d7d9694-n76q9_33e4097c-6799-48a6-9774-d01fbb0adc4e/registry/0.log"
Jul 11 03:43:52.728930 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.728865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:52.789955 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:52.789897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:53.081434 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:53.081401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:53.081614 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:53.081466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:53.081614 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:53.081555    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 11 03:43:53.081614 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:53.081564    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:53.081716 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:53.081619    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:57.081606224 +0000 UTC m=+81.951052765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:53.081716 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:53.081633    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:57.081625949 +0000 UTC m=+81.951072491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : configmap "kube-rbac-proxy" not found
Jul 11 03:43:53.102724 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:53.102677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:53.154881 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:53.154857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:53.200587 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:53.200565    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nswmp_eb2d92a6-4c87-47ae-af3f-8634e40903e6/node-ca/0.log"
Jul 11 03:43:54.199992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:54.199959    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxg8q_14abdefe-12c5-4266-a648-06bd2fb711e1/serve-healthcheck-canary/0.log"
Jul 11 03:43:54.872484 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:54.872447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:54.974092 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:54.974058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:55.022411 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:55.022380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:55.691083 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:55.691052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:55.871621 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:55.871596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:55.974811 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:55.974725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:57.112201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:57.112074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:57.112201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:57.112187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:43:57.112607 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:57.112202    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 11 03:43:57.112607 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:57.112265    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:44:05.112246533 +0000 UTC m=+89.981693078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : configmap "kube-rbac-proxy" not found
Jul 11 03:43:57.112607 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:57.112299    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:57.112607 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:43:57.112337    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls podName:c26922ba-a360-408d-a2ac-e150405b17b8 nodeName:}" failed. No retries permitted until 2026-07-11 03:44:05.112325809 +0000 UTC m=+89.981772366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-crc28" (UID: "c26922ba-a360-408d-a2ac-e150405b17b8") : secret "insights-runtime-extractor-tls" not found
Jul 11 03:43:57.894769 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:57.894732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.014570 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.014497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.121135 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.121039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.189651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.189583    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jqdmt"]
Jul 11 03:43:58.191728 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.191714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.193964 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.193943    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 11 03:43:58.194346 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.194331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 11 03:43:58.194432 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.194332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mqg6h\""
Jul 11 03:43:58.194948 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.194931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 11 03:43:58.195039 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.194931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 11 03:43:58.195039 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.194975    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 11 03:43:58.195039 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.195031    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 11 03:43:58.218579 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.218551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.292373 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.292340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.322807 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.322774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf126a7e-3174-4727-a28e-27e3e6985488-metrics-client-ca\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.322992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.322827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-textfile\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.322992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.322847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-tls\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.322992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.322950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjb6q\" (UniqueName: \"kubernetes.io/projected/cf126a7e-3174-4727-a28e-27e3e6985488-kube-api-access-rjb6q\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.322992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.322986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-sys\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.323201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.323062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-root\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.323201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.323118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.323201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.323147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-wtmp\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.323313 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.323210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.395542 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.395505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.424170 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-root\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424322 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424322 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-wtmp\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424322 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424322 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf126a7e-3174-4727-a28e-27e3e6985488-metrics-client-ca\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424322 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-root\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-textfile\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-tls\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-wtmp\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjb6q\" (UniqueName: \"kubernetes.io/projected/cf126a7e-3174-4727-a28e-27e3e6985488-kube-api-access-rjb6q\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-sys\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf126a7e-3174-4727-a28e-27e3e6985488-sys\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424840 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-textfile\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.424909 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf126a7e-3174-4727-a28e-27e3e6985488-metrics-client-ca\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.425063 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.424956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.426738 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.426710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.426836 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.426798    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf126a7e-3174-4727-a28e-27e3e6985488-node-exporter-tls\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.432058 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.432035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjb6q\" (UniqueName: \"kubernetes.io/projected/cf126a7e-3174-4727-a28e-27e3e6985488-kube-api-access-rjb6q\") pod \"node-exporter-jqdmt\" (UID: \"cf126a7e-3174-4727-a28e-27e3e6985488\") " pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.497285 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.497247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.501385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.501369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqdmt"
Jul 11 03:43:58.511520 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:43:58.511492    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf126a7e_3174_4727_a28e_27e3e6985488.slice/crio-f9661b127936f3d351d547fc49376277ab12cb4cb4921ed519155ecee89b5272 WatchSource:0}: Error finding container f9661b127936f3d351d547fc49376277ab12cb4cb4921ed519155ecee89b5272: Status 404 returned error can't find the container with id f9661b127936f3d351d547fc49376277ab12cb4cb4921ed519155ecee89b5272
Jul 11 03:43:58.594384 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.594350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.692307 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.692279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:58.755790 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:58.755712    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:59.113955 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:59.113880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqdmt" event={"ID":"cf126a7e-3174-4727-a28e-27e3e6985488","Type":"ContainerStarted","Data":"f9661b127936f3d351d547fc49376277ab12cb4cb4921ed519155ecee89b5272"}
Jul 11 03:43:59.143606 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:59.143559    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:59.222728 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:59.222689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:43:59.785903 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:43:59.785874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:00.088834 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:00.088761    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zzhr2"
Jul 11 03:44:00.121870 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:00.121839    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf126a7e-3174-4727-a28e-27e3e6985488" containerID="85d628b5f55954ca6ff2a1bd2fb3a2b52aa3371d4bbdc9c105f3baa6576222fb" exitCode=0
Jul 11 03:44:00.122022 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:00.121889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqdmt" event={"ID":"cf126a7e-3174-4727-a28e-27e3e6985488","Type":"ContainerDied","Data":"85d628b5f55954ca6ff2a1bd2fb3a2b52aa3371d4bbdc9c105f3baa6576222fb"}
Jul 11 03:44:01.112639 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.112609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:01.127523 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.127495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqdmt" event={"ID":"cf126a7e-3174-4727-a28e-27e3e6985488","Type":"ContainerStarted","Data":"60986b769e59124f99e0faa060653e00c0b83a6d6d03b84c2a51519edd717daf"}
Jul 11 03:44:01.127672 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.127544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqdmt" event={"ID":"cf126a7e-3174-4727-a28e-27e3e6985488","Type":"ContainerStarted","Data":"6ba40ea3b9d1ed81899136002169b3ab341c37370e3829c1831cdc6b9ba7cccd"}
Jul 11 03:44:01.151943 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.151898    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jqdmt" podStartSLOduration=2.127326196 podStartE2EDuration="3.151881942s" totalImagesPullingTime="951.336512ms" totalInitContainerRuntime="73.219234ms" isStatefulPod=true podCreationTimestamp="2026-07-11 03:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:01.151019584 +0000 UTC m=+86.020466149" watchObservedRunningTime="2026-07-11 03:44:01.151881942 +0000 UTC m=+86.021328505"
Jul 11 03:44:01.213565 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.213535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:01.313315 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.313283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:01.413231 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.413160    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:01.513384 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:01.513351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:02.911949 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:02.911903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:02.968872 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:02.968839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:03.235552 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:03.235521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:03.334191 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:03.334157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:03.834889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:03.834856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:03.937486 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:03.937451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:03.987275 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:03.987243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:04.124187 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:04.124068    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585d7d9694-n76q9"]
Jul 11 03:44:04.128399 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:04.128377    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:44:04.278491 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:04.278456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:04.378375 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:04.378279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:04.445207 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:04.445172    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:05.180850 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.180813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:44:05.181366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.180890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:44:05.181366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.181361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c26922ba-a360-408d-a2ac-e150405b17b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:44:05.183465 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.183425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c26922ba-a360-408d-a2ac-e150405b17b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-crc28\" (UID: \"c26922ba-a360-408d-a2ac-e150405b17b8\") " pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:44:05.207785 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.207763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:05.266845 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.266815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-crc28"
Jul 11 03:44:05.312929 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.312899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:05.401007 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:05.400977    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-crc28"]
Jul 11 03:44:05.404052 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:44:05.404018    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26922ba_a360_408d_a2ac_e150405b17b8.slice/crio-b461db3236af5becaa74559b9743add330eff7f97744cbca84b64ea31c20ddcd WatchSource:0}: Error finding container b461db3236af5becaa74559b9743add330eff7f97744cbca84b64ea31c20ddcd: Status 404 returned error can't find the container with id b461db3236af5becaa74559b9743add330eff7f97744cbca84b64ea31c20ddcd
Jul 11 03:44:06.147965 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:06.147917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crc28" event={"ID":"c26922ba-a360-408d-a2ac-e150405b17b8","Type":"ContainerStarted","Data":"b461db3236af5becaa74559b9743add330eff7f97744cbca84b64ea31c20ddcd"}
Jul 11 03:44:07.306225 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:07.306197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:07.406453 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:07.406411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:10.163953 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:10.163874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crc28" event={"ID":"c26922ba-a360-408d-a2ac-e150405b17b8","Type":"ContainerStarted","Data":"b342bdb9a57b4901d13110266888fe824760438228e047a20a730d816637e44f"}
Jul 11 03:44:10.163953 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:10.163942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crc28" event={"ID":"c26922ba-a360-408d-a2ac-e150405b17b8","Type":"ContainerStarted","Data":"7900ecfd173dce0bfe6e2ab64f9b0adea9080451192d39078e3f6c0cacabe2d8"}
Jul 11 03:44:12.104651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:12.104611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:12.159330 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:12.159303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:13.040676 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:13.040642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g28hp"
Jul 11 03:44:13.176723 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:13.176682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-crc28" event={"ID":"c26922ba-a360-408d-a2ac-e150405b17b8","Type":"ContainerStarted","Data":"86c219967027385e9a816c12c6153d5ffa9322b9e48fb87841ef75f76a537472"}
Jul 11 03:44:13.192712 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:13.192669    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-crc28" podStartSLOduration=17.366241669 podStartE2EDuration="24.192651661s" totalImagesPullingTime="6.826409992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:43:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:13.192175879 +0000 UTC m=+98.061622444" watchObservedRunningTime="2026-07-11 03:44:13.192651661 +0000 UTC m=+98.062098228"
Jul 11 03:44:14.103858 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:14.103825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:14.205149 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:14.205119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:14.305267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:14.305235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:16.427854 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:16.427818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:16.528663 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:16.528630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:17.767559 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:17.767521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:17.819281 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:17.819249    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:23.478782 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.478750    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8677d6954d-97h75"]
Jul 11 03:44:23.554464 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.554430    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8677d6954d-97h75"]
Jul 11 03:44:23.554639 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.554551    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.557067 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.557046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6nww6\""
Jul 11 03:44:23.557067 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.557055    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 11 03:44:23.558000 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.557979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 11 03:44:23.558000 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.557994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 11 03:44:23.558634 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.558612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 11 03:44:23.558706 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.558640    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 11 03:44:23.559222 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.559206    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 11 03:44:23.559815 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.559797    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 11 03:44:23.567138 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.567120    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 11 03:44:23.602540 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2cc\" (UniqueName: \"kubernetes.io/projected/3b51c09d-38f9-4777-9fed-bfbe96fa481f-kube-api-access-9h2cc\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.602540 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-serving-cert\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.602792 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-config\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.602792 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-oauth-serving-cert\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.602792 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-trusted-ca-bundle\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.602792 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-service-ca\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.602792 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.602775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-oauth-config\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704091 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-oauth-serving-cert\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-trusted-ca-bundle\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-service-ca\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-oauth-config\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2cc\" (UniqueName: \"kubernetes.io/projected/3b51c09d-38f9-4777-9fed-bfbe96fa481f-kube-api-access-9h2cc\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-serving-cert\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-config\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.704946 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.704917    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-oauth-serving-cert\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.705073 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.705037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-config\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.705073 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.705058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-service-ca\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.705216 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.705075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-trusted-ca-bundle\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.707343 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.707318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-oauth-config\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.707530 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.707509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-serving-cert\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.711835 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.711810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2cc\" (UniqueName: \"kubernetes.io/projected/3b51c09d-38f9-4777-9fed-bfbe96fa481f-kube-api-access-9h2cc\") pod \"console-8677d6954d-97h75\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") " pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:23.867549 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:23.867517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:24.005408 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:24.005342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8677d6954d-97h75"]
Jul 11 03:44:24.008007 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:44:24.007979    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b51c09d_38f9_4777_9fed_bfbe96fa481f.slice/crio-559039ef27fa65d74c5f0d3457e6cdc4f1acfd3bfe5895d1dad6fc3620de9ce2 WatchSource:0}: Error finding container 559039ef27fa65d74c5f0d3457e6cdc4f1acfd3bfe5895d1dad6fc3620de9ce2: Status 404 returned error can't find the container with id 559039ef27fa65d74c5f0d3457e6cdc4f1acfd3bfe5895d1dad6fc3620de9ce2
Jul 11 03:44:24.215066 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:24.214973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677d6954d-97h75" event={"ID":"3b51c09d-38f9-4777-9fed-bfbe96fa481f","Type":"ContainerStarted","Data":"559039ef27fa65d74c5f0d3457e6cdc4f1acfd3bfe5895d1dad6fc3620de9ce2"}
Jul 11 03:44:27.227260 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:27.227227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677d6954d-97h75" event={"ID":"3b51c09d-38f9-4777-9fed-bfbe96fa481f","Type":"ContainerStarted","Data":"2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728"}
Jul 11 03:44:27.245288 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:27.245244    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8677d6954d-97h75" podStartSLOduration=1.600402235 podStartE2EDuration="4.245227665s" totalImagesPullingTime="2.64482543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:27.244776078 +0000 UTC m=+112.114222642" watchObservedRunningTime="2026-07-11 03:44:27.245227665 +0000 UTC m=+112.114674228"
Jul 11 03:44:27.727789 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:27.727739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:27.782142 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:27.782087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:29.144889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.144825    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-585d7d9694-n76q9" podUID="33e4097c-6799-48a6-9774-d01fbb0adc4e" containerName="registry" containerID="cri-o://14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040" gracePeriod=30
Jul 11 03:44:29.376601 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.376580    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:44:29.445985 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.445897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-image-registry-private-configuration\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.445985 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.445949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-certificates\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.445985 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.445970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33e4097c-6799-48a6-9774-d01fbb0adc4e-ca-trust-extracted\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.446267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446092    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-kube-api-access-lsl88\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.446267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446155    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.446267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446182    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-trusted-ca\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.446267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446200    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-bound-sa-token\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.446267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446257    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-installation-pull-secrets\") pod \"33e4097c-6799-48a6-9774-d01fbb0adc4e\" (UID: \"33e4097c-6799-48a6-9774-d01fbb0adc4e\") "
Jul 11 03:44:29.446512 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446342    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:44:29.446667 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446632    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-certificates\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.446797 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.446676    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:44:29.448962 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.448930    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:44:29.449060 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.448960    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:44:29.449060 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.449023    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-kube-api-access-lsl88" (OuterVolumeSpecName: "kube-api-access-lsl88") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "kube-api-access-lsl88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:44:29.449166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.449115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:44:29.449201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.449159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:44:29.454535 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.454515    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e4097c-6799-48a6-9774-d01fbb0adc4e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "33e4097c-6799-48a6-9774-d01fbb0adc4e" (UID: "33e4097c-6799-48a6-9774-d01fbb0adc4e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 03:44:29.547900 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547861    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-kube-api-access-lsl88\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.547900 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547892    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-registry-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.547900 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547901    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33e4097c-6799-48a6-9774-d01fbb0adc4e-trusted-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.547900 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547912    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e4097c-6799-48a6-9774-d01fbb0adc4e-bound-sa-token\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.548177 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547920    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-installation-pull-secrets\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.548177 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547929    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33e4097c-6799-48a6-9774-d01fbb0adc4e-image-registry-private-configuration\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:29.548177 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:29.547937    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33e4097c-6799-48a6-9774-d01fbb0adc4e-ca-trust-extracted\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:44:30.239607 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.239571    2582 generic.go:356] "Generic (PLEG): container finished" podID="33e4097c-6799-48a6-9774-d01fbb0adc4e" containerID="14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040" exitCode=0
Jul 11 03:44:30.240011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.239616    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585d7d9694-n76q9"
Jul 11 03:44:30.240011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.239615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585d7d9694-n76q9" event={"ID":"33e4097c-6799-48a6-9774-d01fbb0adc4e","Type":"ContainerDied","Data":"14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040"}
Jul 11 03:44:30.240011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.239736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585d7d9694-n76q9" event={"ID":"33e4097c-6799-48a6-9774-d01fbb0adc4e","Type":"ContainerDied","Data":"bcd2becef3b984b153c2ea57dc23e9bb19b33a3744d127e1b1d90aec57feafd1"}
Jul 11 03:44:30.240011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.239776    2582 scope.go:117] "RemoveContainer" containerID="14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040"
Jul 11 03:44:30.248401 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.248384    2582 scope.go:117] "RemoveContainer" containerID="14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040"
Jul 11 03:44:30.248643 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:44:30.248626    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040\": container with ID starting with 14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040 not found: ID does not exist" containerID="14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040"
Jul 11 03:44:30.248688 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.248650    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040"} err="failed to get container status \"14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040\": rpc error: code = NotFound desc = could not find container \"14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040\": container with ID starting with 14eccf5eba99cd16d9c8d47604f2efd213bf62218ac7e74b6bfdc4ea09f33040 not found: ID does not exist"
Jul 11 03:44:30.256721 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.256694    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585d7d9694-n76q9"]
Jul 11 03:44:30.262909 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:30.262886    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-585d7d9694-n76q9"]
Jul 11 03:44:31.119277 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:31.119242    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:31.174548 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:31.174511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:31.644695 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:31.644666    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e4097c-6799-48a6-9774-d01fbb0adc4e" path="/var/lib/kubelet/pods/33e4097c-6799-48a6-9774-d01fbb0adc4e/volumes"
Jul 11 03:44:32.750778 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:32.750746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:32.801857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:32.801829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:33.868132 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:33.868083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:33.868518 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:33.868147    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:33.873228 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:33.873207    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:34.257453 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:34.257427    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:44:34.396417 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:34.396387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:34.502154 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:34.502125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:34.602841 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:34.602766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:35.610354 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:35.610323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:36.261849 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:36.261818    2582 generic.go:356] "Generic (PLEG): container finished" podID="92473190-1f60-4e48-9669-94df6487b33d" containerID="3dceb6b1940926c9e4d8b4c6a6152d24d2ea56dd814fd1fcb73a7d70e584688e" exitCode=0
Jul 11 03:44:36.262023 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:36.261891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n" event={"ID":"92473190-1f60-4e48-9669-94df6487b33d","Type":"ContainerDied","Data":"3dceb6b1940926c9e4d8b4c6a6152d24d2ea56dd814fd1fcb73a7d70e584688e"}
Jul 11 03:44:36.262244 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:36.262228    2582 scope.go:117] "RemoveContainer" containerID="3dceb6b1940926c9e4d8b4c6a6152d24d2ea56dd814fd1fcb73a7d70e584688e"
Jul 11 03:44:37.267176 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:37.267135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-fgh2n" event={"ID":"92473190-1f60-4e48-9669-94df6487b33d","Type":"ContainerStarted","Data":"cf0e55eaea0801ba3fd42522398ae3b7c34926690acd50094064abbb2ddf40d0"}
Jul 11 03:44:41.147907 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:41.147872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:41.199494 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:41.199466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:47.758110 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:47.758073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:44:47.813366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:44:47.813333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:14.280820 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:14.280787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:14.381730 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:14.381686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:19.583589 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:19.583542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:19.639870 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:19.639842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:26.150076 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:26.150037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:26.203239 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:26.203203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:32.760080 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:32.760044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:32.857239 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:32.857214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:32.925950 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:32.925920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:38.428327 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:38.428270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:39.709036 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:39.708997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:50.989382 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:50.989350    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8677d6954d-97h75"]
Jul 11 03:45:51.077305 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:51.077266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:45:51.177900 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:45:51.177866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:09.859429 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:09.859392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:09.957475 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:09.957440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:16.010681 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.010610    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8677d6954d-97h75" podUID="3b51c09d-38f9-4777-9fed-bfbe96fa481f" containerName="console" containerID="cri-o://2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728" gracePeriod=15
Jul 11 03:46:16.254328 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.254306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:46:16.331447 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-trusted-ca-bundle\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331618 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331450    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-config\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331618 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331481    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-serving-cert\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331618 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331511    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-oauth-serving-cert\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331618 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331581    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-service-ca\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331618 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331613    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h2cc\" (UniqueName: \"kubernetes.io/projected/3b51c09d-38f9-4777-9fed-bfbe96fa481f-kube-api-access-9h2cc\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331865 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:46:16.331865 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331655    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-oauth-config\") pod \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\" (UID: \"3b51c09d-38f9-4777-9fed-bfbe96fa481f\") "
Jul 11 03:46:16.331957 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331888    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-trusted-ca-bundle\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.331957 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-config" (OuterVolumeSpecName: "console-config") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:46:16.332124 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.331947    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:46:16.332124 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.332043    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:46:16.333956 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.333923    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:46:16.333956 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.333942    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:46:16.334213 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.333980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b51c09d-38f9-4777-9fed-bfbe96fa481f-kube-api-access-9h2cc" (OuterVolumeSpecName: "kube-api-access-9h2cc") pod "3b51c09d-38f9-4777-9fed-bfbe96fa481f" (UID: "3b51c09d-38f9-4777-9fed-bfbe96fa481f"). InnerVolumeSpecName "kube-api-access-9h2cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:46:16.432617 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.432578    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h2cc\" (UniqueName: \"kubernetes.io/projected/3b51c09d-38f9-4777-9fed-bfbe96fa481f-kube-api-access-9h2cc\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.432617 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.432608    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-oauth-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.432617 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.432617    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.432617 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.432626    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b51c09d-38f9-4777-9fed-bfbe96fa481f-console-serving-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.432865 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.432635    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-oauth-serving-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.432865 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.432643    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b51c09d-38f9-4777-9fed-bfbe96fa481f-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:46:16.603703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.603617    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b51c09d-38f9-4777-9fed-bfbe96fa481f" containerID="2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728" exitCode=0
Jul 11 03:46:16.603703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.603665    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677d6954d-97h75"
Jul 11 03:46:16.603703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.603671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677d6954d-97h75" event={"ID":"3b51c09d-38f9-4777-9fed-bfbe96fa481f","Type":"ContainerDied","Data":"2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728"}
Jul 11 03:46:16.603936 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.603714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677d6954d-97h75" event={"ID":"3b51c09d-38f9-4777-9fed-bfbe96fa481f","Type":"ContainerDied","Data":"559039ef27fa65d74c5f0d3457e6cdc4f1acfd3bfe5895d1dad6fc3620de9ce2"}
Jul 11 03:46:16.603936 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.603733    2582 scope.go:117] "RemoveContainer" containerID="2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728"
Jul 11 03:46:16.612021 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.612001    2582 scope.go:117] "RemoveContainer" containerID="2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728"
Jul 11 03:46:16.612292 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:46:16.612274    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728\": container with ID starting with 2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728 not found: ID does not exist" containerID="2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728"
Jul 11 03:46:16.612343 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.612301    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728"} err="failed to get container status \"2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728\": rpc error: code = NotFound desc = could not find container \"2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728\": container with ID starting with 2be51f6475f00b217c3052211d0aec9f6842f2fab5933096f1dd9a8792871728 not found: ID does not exist"
Jul 11 03:46:16.623344 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.623320    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8677d6954d-97h75"]
Jul 11 03:46:16.627183 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:16.627164    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8677d6954d-97h75"]
Jul 11 03:46:17.643799 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:17.643769    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b51c09d-38f9-4777-9fed-bfbe96fa481f" path="/var/lib/kubelet/pods/3b51c09d-38f9-4777-9fed-bfbe96fa481f/volumes"
Jul 11 03:46:27.425228 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:27.425187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:31.425649 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:31.425611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:32.247448 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:32.247407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:34.885624 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:34.885592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:35.039069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:35.039036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:35.090288 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:35.090254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:36.559888 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:36.559856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:36.676509 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:36.676477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:38.890406 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:38.890370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:39.304746 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:39.304696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:41.831826 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:41.831768    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:41.932216 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:41.932187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:42.510742 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:42.510704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:44.075884 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.075851    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rv4qb"]
Jul 11 03:46:44.076389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.076194    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e4097c-6799-48a6-9774-d01fbb0adc4e" containerName="registry"
Jul 11 03:46:44.076389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.076212    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e4097c-6799-48a6-9774-d01fbb0adc4e" containerName="registry"
Jul 11 03:46:44.076389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.076229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b51c09d-38f9-4777-9fed-bfbe96fa481f" containerName="console"
Jul 11 03:46:44.076389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.076237    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b51c09d-38f9-4777-9fed-bfbe96fa481f" containerName="console"
Jul 11 03:46:44.076389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.076303    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b51c09d-38f9-4777-9fed-bfbe96fa481f" containerName="console"
Jul 11 03:46:44.076389 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.076316    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e4097c-6799-48a6-9774-d01fbb0adc4e" containerName="registry"
Jul 11 03:46:44.080287 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.080267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.082750 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.082729    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 11 03:46:44.082846 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.082793    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mkkjz\""
Jul 11 03:46:44.083636 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.083620    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 11 03:46:44.088433 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.088403    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rv4qb"]
Jul 11 03:46:44.236250 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.236216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6vv\" (UniqueName: \"kubernetes.io/projected/53d49991-a0f4-43e7-a1bb-6de7cae54378-kube-api-access-fx6vv\") pod \"cert-manager-webhook-6486ffd997-rv4qb\" (UID: \"53d49991-a0f4-43e7-a1bb-6de7cae54378\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.236416 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.236271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53d49991-a0f4-43e7-a1bb-6de7cae54378-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rv4qb\" (UID: \"53d49991-a0f4-43e7-a1bb-6de7cae54378\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.337366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.337294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53d49991-a0f4-43e7-a1bb-6de7cae54378-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rv4qb\" (UID: \"53d49991-a0f4-43e7-a1bb-6de7cae54378\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.337366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.337348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6vv\" (UniqueName: \"kubernetes.io/projected/53d49991-a0f4-43e7-a1bb-6de7cae54378-kube-api-access-fx6vv\") pod \"cert-manager-webhook-6486ffd997-rv4qb\" (UID: \"53d49991-a0f4-43e7-a1bb-6de7cae54378\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.344964 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.344935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6vv\" (UniqueName: \"kubernetes.io/projected/53d49991-a0f4-43e7-a1bb-6de7cae54378-kube-api-access-fx6vv\") pod \"cert-manager-webhook-6486ffd997-rv4qb\" (UID: \"53d49991-a0f4-43e7-a1bb-6de7cae54378\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.345066 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.344938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53d49991-a0f4-43e7-a1bb-6de7cae54378-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rv4qb\" (UID: \"53d49991-a0f4-43e7-a1bb-6de7cae54378\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.389980 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.389941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:44.514875 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.514846    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rv4qb"]
Jul 11 03:46:44.517771 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:46:44.517749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53d49991_a0f4_43e7_a1bb_6de7cae54378.slice/crio-b91666fecf126f14f46576775f51f7ba055cc92b99f38cd510374fb116d7f8d7 WatchSource:0}: Error finding container b91666fecf126f14f46576775f51f7ba055cc92b99f38cd510374fb116d7f8d7: Status 404 returned error can't find the container with id b91666fecf126f14f46576775f51f7ba055cc92b99f38cd510374fb116d7f8d7
Jul 11 03:46:44.698045 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:44.697957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb" event={"ID":"53d49991-a0f4-43e7-a1bb-6de7cae54378","Type":"ContainerStarted","Data":"b91666fecf126f14f46576775f51f7ba055cc92b99f38cd510374fb116d7f8d7"}
Jul 11 03:46:46.105118 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:46.105065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:47.710616 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:47.710526    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb" event={"ID":"53d49991-a0f4-43e7-a1bb-6de7cae54378","Type":"ContainerStarted","Data":"086034e3a70bc3495519a2ec7dcabef985734c02549cdd45b92fe05b242b5843"}
Jul 11 03:46:47.710994 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:47.710616    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:47.725065 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:47.725023    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb" podStartSLOduration=0.953601187 podStartE2EDuration="3.725011617s" totalImagesPullingTime="2.77141043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:46:47.724037317 +0000 UTC m=+252.593483875" watchObservedRunningTime="2026-07-11 03:46:47.725011617 +0000 UTC m=+252.594458180"
Jul 11 03:46:47.845620 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:47.845586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:47.939420 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:47.939389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:48.946432 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:48.946398    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ztwns"]
Jul 11 03:46:48.949554 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:48.949536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:48.951956 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:48.951934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jfvdb\""
Jul 11 03:46:48.959050 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:48.959029    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ztwns"]
Jul 11 03:46:49.073392 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.073355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pps\" (UniqueName: \"kubernetes.io/projected/c83d311d-cfba-4370-b8d9-cba8cf599e3f-kube-api-access-l8pps\") pod \"cert-manager-55b4d954d5-ztwns\" (UID: \"c83d311d-cfba-4370-b8d9-cba8cf599e3f\") " pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.073579 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.073417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c83d311d-cfba-4370-b8d9-cba8cf599e3f-bound-sa-token\") pod \"cert-manager-55b4d954d5-ztwns\" (UID: \"c83d311d-cfba-4370-b8d9-cba8cf599e3f\") " pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.174484 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.174452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pps\" (UniqueName: \"kubernetes.io/projected/c83d311d-cfba-4370-b8d9-cba8cf599e3f-kube-api-access-l8pps\") pod \"cert-manager-55b4d954d5-ztwns\" (UID: \"c83d311d-cfba-4370-b8d9-cba8cf599e3f\") " pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.174666 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.174518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c83d311d-cfba-4370-b8d9-cba8cf599e3f-bound-sa-token\") pod \"cert-manager-55b4d954d5-ztwns\" (UID: \"c83d311d-cfba-4370-b8d9-cba8cf599e3f\") " pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.182922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.182898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pps\" (UniqueName: \"kubernetes.io/projected/c83d311d-cfba-4370-b8d9-cba8cf599e3f-kube-api-access-l8pps\") pod \"cert-manager-55b4d954d5-ztwns\" (UID: \"c83d311d-cfba-4370-b8d9-cba8cf599e3f\") " pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.183042 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.182902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c83d311d-cfba-4370-b8d9-cba8cf599e3f-bound-sa-token\") pod \"cert-manager-55b4d954d5-ztwns\" (UID: \"c83d311d-cfba-4370-b8d9-cba8cf599e3f\") " pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.257924 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.257884    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ztwns"
Jul 11 03:46:49.386918 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.386887    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ztwns"]
Jul 11 03:46:49.389902 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:46:49.389872    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83d311d_cfba_4370_b8d9_cba8cf599e3f.slice/crio-fcb11385113b6e3e9e71c9b3d92072869c6c4458018006c141204b9d931a00c1 WatchSource:0}: Error finding container fcb11385113b6e3e9e71c9b3d92072869c6c4458018006c141204b9d931a00c1: Status 404 returned error can't find the container with id fcb11385113b6e3e9e71c9b3d92072869c6c4458018006c141204b9d931a00c1
Jul 11 03:46:49.718450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.718366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ztwns" event={"ID":"c83d311d-cfba-4370-b8d9-cba8cf599e3f","Type":"ContainerStarted","Data":"83cea9ce8ee1c76b6cb1a50a2e03b200d5bfec76163038a360575cb16fb8e073"}
Jul 11 03:46:49.718450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.718405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ztwns" event={"ID":"c83d311d-cfba-4370-b8d9-cba8cf599e3f","Type":"ContainerStarted","Data":"fcb11385113b6e3e9e71c9b3d92072869c6c4458018006c141204b9d931a00c1"}
Jul 11 03:46:49.732597 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:49.732545    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ztwns" podStartSLOduration=1.732526163 podStartE2EDuration="1.732526163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:46:49.732189479 +0000 UTC m=+254.601636045" watchObservedRunningTime="2026-07-11 03:46:49.732526163 +0000 UTC m=+254.601972729"
Jul 11 03:46:53.717131 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:53.717079    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rv4qb"
Jul 11 03:46:54.937850 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:54.937816    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:55.038215 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:55.038186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:46:56.816327 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:46:56.816247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:00.952604 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:00.952572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:01.005974 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:01.005942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:09.635680 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:09.635644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:09.734391 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:09.734359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:10.260064 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.260029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:10.459510 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.459475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:10.556335 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.556249    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:10.608463 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.608426    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"]
Jul 11 03:47:10.611613 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.611584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.615196 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.615163    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 11 03:47:10.615346 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.615199    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 11 03:47:10.615346 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.615236    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2vrhl\""
Jul 11 03:47:10.615487 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.615436    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 11 03:47:10.615652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.615640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 11 03:47:10.617836 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.617815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7bd2c6b-7412-436a-a48c-a37b239480b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.617941 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.617854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7bd2c6b-7412-436a-a48c-a37b239480b8-tmp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.617941 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.617882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7bd2c6b-7412-436a-a48c-a37b239480b8-manifests\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.617941 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.617905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7bd2c6b-7412-436a-a48c-a37b239480b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.617941 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.617938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznnp\" (UniqueName: \"kubernetes.io/projected/c7bd2c6b-7412-436a-a48c-a37b239480b8-kube-api-access-vznnp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.633525 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.633501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"]
Jul 11 03:47:10.718857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.718821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznnp\" (UniqueName: \"kubernetes.io/projected/c7bd2c6b-7412-436a-a48c-a37b239480b8-kube-api-access-vznnp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.718857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.718862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7bd2c6b-7412-436a-a48c-a37b239480b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.719298 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.718900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7bd2c6b-7412-436a-a48c-a37b239480b8-tmp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.719298 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.718920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7bd2c6b-7412-436a-a48c-a37b239480b8-manifests\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.719298 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.718936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7bd2c6b-7412-436a-a48c-a37b239480b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.719448 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.719425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7bd2c6b-7412-436a-a48c-a37b239480b8-manifests\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.721306 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.721277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7bd2c6b-7412-436a-a48c-a37b239480b8-tmp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.721588 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.721571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7bd2c6b-7412-436a-a48c-a37b239480b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.721588 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.721583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7bd2c6b-7412-436a-a48c-a37b239480b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.731338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.731312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznnp\" (UniqueName: \"kubernetes.io/projected/c7bd2c6b-7412-436a-a48c-a37b239480b8-kube-api-access-vznnp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-b8mwq\" (UID: \"c7bd2c6b-7412-436a-a48c-a37b239480b8\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:10.788237 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.788203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:10.907615 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.907526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:10.921658 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:10.921633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:11.011313 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:11.011277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:11.056683 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:11.056645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"]
Jul 11 03:47:11.059758 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:47:11.059729    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bd2c6b_7412_436a_a48c_a37b239480b8.slice/crio-1343aef43868215d3efa56f6cb3d453b5e54d4efb37cb50fc4c6ca6b3082c5dc WatchSource:0}: Error finding container 1343aef43868215d3efa56f6cb3d453b5e54d4efb37cb50fc4c6ca6b3082c5dc: Status 404 returned error can't find the container with id 1343aef43868215d3efa56f6cb3d453b5e54d4efb37cb50fc4c6ca6b3082c5dc
Jul 11 03:47:11.108559 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:11.108528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:11.213179 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:11.213081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:11.316872 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:11.316839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:11.797598 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:11.797542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq" event={"ID":"c7bd2c6b-7412-436a-a48c-a37b239480b8","Type":"ContainerStarted","Data":"1343aef43868215d3efa56f6cb3d453b5e54d4efb37cb50fc4c6ca6b3082c5dc"}
Jul 11 03:47:14.810878 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:14.810840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq" event={"ID":"c7bd2c6b-7412-436a-a48c-a37b239480b8","Type":"ContainerStarted","Data":"9a583cdd5c3b45251896978cea8041f3492015bef8f9da28cd317e75320d6eaa"}
Jul 11 03:47:15.815701 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:15.815670    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7bd2c6b-7412-436a-a48c-a37b239480b8" containerID="9a583cdd5c3b45251896978cea8041f3492015bef8f9da28cd317e75320d6eaa" exitCode=0
Jul 11 03:47:15.816089 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:15.815767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq" event={"ID":"c7bd2c6b-7412-436a-a48c-a37b239480b8","Type":"ContainerDied","Data":"9a583cdd5c3b45251896978cea8041f3492015bef8f9da28cd317e75320d6eaa"}
Jul 11 03:47:16.820801 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:16.820766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq" event={"ID":"c7bd2c6b-7412-436a-a48c-a37b239480b8","Type":"ContainerStarted","Data":"ebdccab71b787ea2961dd1b215aaa021fb28c84998a7131271498c4755b188f8"}
Jul 11 03:47:16.821173 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:16.820895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:16.841864 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:16.841816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq" podStartSLOduration=2.472978511 podStartE2EDuration="6.841802194s" totalImagesPullingTime="3.370556612s" totalInitContainerRuntime="998.267071ms" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:16.840959117 +0000 UTC m=+281.710405681" watchObservedRunningTime="2026-07-11 03:47:16.841802194 +0000 UTC m=+281.711248758"
Jul 11 03:47:21.040358 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:21.040313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:21.148852 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:21.148817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:21.780003 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:21.779970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:21.879703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:21.879673    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:27.827653 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:27.827627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-b8mwq"
Jul 11 03:47:29.368033 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.367990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:29.474291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.474261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:29.774308 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.774276    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:29.941067 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.941035    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"]
Jul 11 03:47:29.944047 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.944031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:29.947540 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.947521    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 11 03:47:29.947660 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.947610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 11 03:47:29.947709 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.947682    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 11 03:47:29.947901 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.947885    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4dhw4\""
Jul 11 03:47:29.947978 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.947961    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 11 03:47:29.964741 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:29.964716    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"]
Jul 11 03:47:30.046050 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.045953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr5d\" (UniqueName: \"kubernetes.io/projected/43a572d7-0033-4845-be73-b514a5121669-kube-api-access-bvr5d\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.046050 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.046017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43a572d7-0033-4845-be73-b514a5121669-tmp\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.046263 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.046124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a572d7-0033-4845-be73-b514a5121669-tls-certs\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.096725 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.096699    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4hfnt"]
Jul 11 03:47:30.099654 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.099639    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.102776 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.102755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 11 03:47:30.102881 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.102790    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-w7rwh\""
Jul 11 03:47:30.105740 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.105710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:30.112636 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.112603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4hfnt"]
Jul 11 03:47:30.147258 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.147218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a572d7-0033-4845-be73-b514a5121669-tls-certs\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.147441 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.147270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr5d\" (UniqueName: \"kubernetes.io/projected/43a572d7-0033-4845-be73-b514a5121669-kube-api-access-bvr5d\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.147441 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.147304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43a572d7-0033-4845-be73-b514a5121669-tmp\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.147441 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.147323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fsf\" (UniqueName: \"kubernetes.io/projected/b29da052-ad41-4ff0-840e-9eccef268909-kube-api-access-99fsf\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.147441 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.147350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b29da052-ad41-4ff0-840e-9eccef268909-cert\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.149789 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.149756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43a572d7-0033-4845-be73-b514a5121669-tmp\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.149898 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.149873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a572d7-0033-4845-be73-b514a5121669-tls-certs\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.158728 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.158698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr5d\" (UniqueName: \"kubernetes.io/projected/43a572d7-0033-4845-be73-b514a5121669-kube-api-access-bvr5d\") pod \"kube-auth-proxy-67cfbd58d5-st9mj\" (UID: \"43a572d7-0033-4845-be73-b514a5121669\") " pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.199007 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.198968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:30.248539 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.248504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fsf\" (UniqueName: \"kubernetes.io/projected/b29da052-ad41-4ff0-840e-9eccef268909-kube-api-access-99fsf\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.248713 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.248549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b29da052-ad41-4ff0-840e-9eccef268909-cert\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.248713 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:47:30.248667    2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 11 03:47:30.248800 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:47:30.248730    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29da052-ad41-4ff0-840e-9eccef268909-cert podName:b29da052-ad41-4ff0-840e-9eccef268909 nodeName:}" failed. No retries permitted until 2026-07-11 03:47:30.748714895 +0000 UTC m=+295.618161436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b29da052-ad41-4ff0-840e-9eccef268909-cert") pod "odh-model-controller-7998c67dc4-4hfnt" (UID: "b29da052-ad41-4ff0-840e-9eccef268909") : secret "odh-model-controller-webhook-cert" not found
Jul 11 03:47:30.253505 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.253484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"
Jul 11 03:47:30.263014 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.262982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fsf\" (UniqueName: \"kubernetes.io/projected/b29da052-ad41-4ff0-840e-9eccef268909-kube-api-access-99fsf\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.300372 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.300294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:30.384730 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.384701    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj"]
Jul 11 03:47:30.387209 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:47:30.387180    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a572d7_0033_4845_be73_b514a5121669.slice/crio-027c5c18a3044858782780228e89ed7133077fd01f0dac973a5a1b6469ddf4f2 WatchSource:0}: Error finding container 027c5c18a3044858782780228e89ed7133077fd01f0dac973a5a1b6469ddf4f2: Status 404 returned error can't find the container with id 027c5c18a3044858782780228e89ed7133077fd01f0dac973a5a1b6469ddf4f2
Jul 11 03:47:30.455670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.455632    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:30.752631 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.752599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b29da052-ad41-4ff0-840e-9eccef268909-cert\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.755133 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.755088    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b29da052-ad41-4ff0-840e-9eccef268909-cert\") pod \"odh-model-controller-7998c67dc4-4hfnt\" (UID: \"b29da052-ad41-4ff0-840e-9eccef268909\") " pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:30.872719 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.872680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj" event={"ID":"43a572d7-0033-4845-be73-b514a5121669","Type":"ContainerStarted","Data":"027c5c18a3044858782780228e89ed7133077fd01f0dac973a5a1b6469ddf4f2"}
Jul 11 03:47:30.981549 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:30.981511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:31.009498 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.009410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:31.051929 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.051896    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"]
Jul 11 03:47:31.055299 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.055276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.058045 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.058011    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 11 03:47:31.058045 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.058011    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 11 03:47:31.058250 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.058049    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bch8v\""
Jul 11 03:47:31.058250 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.058019    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 11 03:47:31.058250 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.058019    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 11 03:47:31.069564 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.069490    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"]
Jul 11 03:47:31.073450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.073425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:31.140901 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.140870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:31.156044 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6590914-5575-4fbc-aed1-b165b0833027-local-certs\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.156221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-cacerts\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.156221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.156385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.156385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-istio-token\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.156385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskqm\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-kube-api-access-fskqm\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.156385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.156328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.161780 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.161689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4hfnt"]
Jul 11 03:47:31.164796 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:47:31.164763    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29da052_ad41_4ff0_840e_9eccef268909.slice/crio-1492fbd5e09b377e41692ada7ac6051f39f1bfb1cbf91f28276393a493a2b266 WatchSource:0}: Error finding container 1492fbd5e09b377e41692ada7ac6051f39f1bfb1cbf91f28276393a493a2b266: Status 404 returned error can't find the container with id 1492fbd5e09b377e41692ada7ac6051f39f1bfb1cbf91f28276393a493a2b266
Jul 11 03:47:31.257408 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6590914-5575-4fbc-aed1-b165b0833027-local-certs\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.257408 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-cacerts\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.257651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.257651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.257651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-istio-token\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.257651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskqm\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-kube-api-access-fskqm\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.257651 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.257650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.258227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.258167    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.259985 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.259928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6590914-5575-4fbc-aed1-b165b0833027-local-certs\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.260132 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.260115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.260365 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.260345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-cacerts\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.260424 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.260362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.265635 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.265612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-istio-token\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.265992 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.265976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskqm\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-kube-api-access-fskqm\") pod \"istiod-openshift-gateway-7574459656-djcsm\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.365897 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.365857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:31.525260 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.525237    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"]
Jul 11 03:47:31.526713 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:47:31.526689    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6590914_5575_4fbc_aed1_b165b0833027.slice/crio-374a7a564c417cbc4309731c841649bbe82c70b764f20d48c00fcc3ef26b253a WatchSource:0}: Error finding container 374a7a564c417cbc4309731c841649bbe82c70b764f20d48c00fcc3ef26b253a: Status 404 returned error can't find the container with id 374a7a564c417cbc4309731c841649bbe82c70b764f20d48c00fcc3ef26b253a
Jul 11 03:47:31.880632 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.880545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" event={"ID":"f6590914-5575-4fbc-aed1-b165b0833027","Type":"ContainerStarted","Data":"374a7a564c417cbc4309731c841649bbe82c70b764f20d48c00fcc3ef26b253a"}
Jul 11 03:47:31.882885 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:31.882855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt" event={"ID":"b29da052-ad41-4ff0-840e-9eccef268909","Type":"ContainerStarted","Data":"1492fbd5e09b377e41692ada7ac6051f39f1bfb1cbf91f28276393a493a2b266"}
Jul 11 03:47:32.006176 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:32.006140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:35.673461 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.673433    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 11 03:47:35.678330 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.678272    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:47:35.678410 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.678353    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:47:35.691706 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.691680    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:47:35.691809 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.691680    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:47:35.695413 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.695391    2582 kubelet.go:1648] "Image garbage collection succeeded"
Jul 11 03:47:35.903253 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.903161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" event={"ID":"f6590914-5575-4fbc-aed1-b165b0833027","Type":"ContainerStarted","Data":"7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b"}
Jul 11 03:47:35.903423 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.903282    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:35.904844 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.904816    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt" event={"ID":"b29da052-ad41-4ff0-840e-9eccef268909","Type":"ContainerStarted","Data":"e811d9adeef62a83070d31173883069e6a46d076e2db956992002612d7663d9c"}
Jul 11 03:47:35.904993 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.904872    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:47:35.906346 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.906323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj" event={"ID":"43a572d7-0033-4845-be73-b514a5121669","Type":"ContainerStarted","Data":"97db71cc44d77235cb043f7cc121d99010bc1aec515bfe0ea3317d010a046a9f"}
Jul 11 03:47:35.922657 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.922606    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" podStartSLOduration=0.773367301 podStartE2EDuration="4.922591783s" totalImagesPullingTime="4.149224482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:35.921321267 +0000 UTC m=+300.790767830" watchObservedRunningTime="2026-07-11 03:47:35.922591783 +0000 UTC m=+300.792038341"
Jul 11 03:47:35.938466 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.938416    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67cfbd58d5-st9mj" podStartSLOduration=1.656535742 podStartE2EDuration="6.938402381s" totalImagesPullingTime="5.281866639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:35.937370143 +0000 UTC m=+300.806816709" watchObservedRunningTime="2026-07-11 03:47:35.938402381 +0000 UTC m=+300.807848946"
Jul 11 03:47:35.953552 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:35.953505    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt" podStartSLOduration=1.440537298 podStartE2EDuration="5.953487813s" totalImagesPullingTime="4.512950515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:35.952146035 +0000 UTC m=+300.821592600" watchObservedRunningTime="2026-07-11 03:47:35.953487813 +0000 UTC m=+300.822934377"
Jul 11 03:47:36.692163 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:36.692126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:36.785604 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:36.785558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:36.911958 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:36.911922    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-djcsm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 11 03:47:36.912131 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:36.911977    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" podUID="f6590914-5575-4fbc-aed1-b165b0833027" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 11 03:47:36.936708 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:36.936676    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:37.043738 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:37.043703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:37.263856 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:37.263825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:39.911960 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:39.911931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:40.410557 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.410524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:40.463118 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.463074    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"]
Jul 11 03:47:40.466351 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.466332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.468956 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.468936    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 11 03:47:40.469761 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.469740    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-c489j\""
Jul 11 03:47:40.476197 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.476173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"]
Jul 11 03:47:40.477922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.477901    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"]
Jul 11 03:47:40.480869 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.480851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:40.483982 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.483960    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 11 03:47:40.484079 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.483978    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6b5jb\""
Jul 11 03:47:40.491961 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.491939    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"]
Jul 11 03:47:40.542290 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.542259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:40.542459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.542295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrkm\" (UniqueName: \"kubernetes.io/projected/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-kube-api-access-7wrkm\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:40.542459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.542326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2nh\" (UniqueName: \"kubernetes.io/projected/ff10d8c5-3d3a-4935-a4f6-d193469e6e6c-kube-api-access-rk2nh\") pod \"kserve-controller-manager-d6f4b65b8-bffn4\" (UID: \"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.542459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.542413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff10d8c5-3d3a-4935-a4f6-d193469e6e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-bffn4\" (UID: \"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.643378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.643441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrkm\" (UniqueName: \"kubernetes.io/projected/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-kube-api-access-7wrkm\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.643484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2nh\" (UniqueName: \"kubernetes.io/projected/ff10d8c5-3d3a-4935-a4f6-d193469e6e6c-kube-api-access-rk2nh\") pod \"kserve-controller-manager-d6f4b65b8-bffn4\" (UID: \"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.643549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff10d8c5-3d3a-4935-a4f6-d193469e6e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-bffn4\" (UID: \"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:47:40.644407    2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:47:40.644475    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-cert podName:9a83a035-1fa4-4f07-9bf4-31be8f6e5661 nodeName:}" failed. No retries permitted until 2026-07-11 03:47:41.144452642 +0000 UTC m=+306.013899191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-cert") pod "llmisvc-controller-manager-777c6bd7d5-vd8l9" (UID: "9a83a035-1fa4-4f07-9bf4-31be8f6e5661") : secret "llmisvc-webhook-server-cert" not found
Jul 11 03:47:40.652868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.647602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff10d8c5-3d3a-4935-a4f6-d193469e6e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-bffn4\" (UID: \"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.666961 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.666895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2nh\" (UniqueName: \"kubernetes.io/projected/ff10d8c5-3d3a-4935-a4f6-d193469e6e6c-kube-api-access-rk2nh\") pod \"kserve-controller-manager-d6f4b65b8-bffn4\" (UID: \"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.667130 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.666961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrkm\" (UniqueName: \"kubernetes.io/projected/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-kube-api-access-7wrkm\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:40.777581 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.777547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:40.919058 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.919032    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"]
Jul 11 03:47:40.921382 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:47:40.921356    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff10d8c5_3d3a_4935_a4f6_d193469e6e6c.slice/crio-4a83b09b25d17c7f7e42040aebe7d0bf4a8ec9e878e525fc36f554c547432756 WatchSource:0}: Error finding container 4a83b09b25d17c7f7e42040aebe7d0bf4a8ec9e878e525fc36f554c547432756: Status 404 returned error can't find the container with id 4a83b09b25d17c7f7e42040aebe7d0bf4a8ec9e878e525fc36f554c547432756
Jul 11 03:47:40.922617 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.922601    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 03:47:40.926964 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:40.926939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4" event={"ID":"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c","Type":"ContainerStarted","Data":"4a83b09b25d17c7f7e42040aebe7d0bf4a8ec9e878e525fc36f554c547432756"}
Jul 11 03:47:41.147204 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.147163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:41.149775 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.149755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a83a035-1fa4-4f07-9bf4-31be8f6e5661-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vd8l9\" (UID: \"9a83a035-1fa4-4f07-9bf4-31be8f6e5661\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:41.391593 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.391556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:41.522627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.522591    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"]
Jul 11 03:47:41.524718 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:47:41.524690    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a83a035_1fa4_4f07_9bf4_31be8f6e5661.slice/crio-b98e3b178baaff114f22924cd125c2082f19581e6b4d8af31cbe029f9198a666 WatchSource:0}: Error finding container b98e3b178baaff114f22924cd125c2082f19581e6b4d8af31cbe029f9198a666: Status 404 returned error can't find the container with id b98e3b178baaff114f22924cd125c2082f19581e6b4d8af31cbe029f9198a666
Jul 11 03:47:41.537932 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.537891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:41.751378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.751343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:41.929566 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.929513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:41.934241 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:41.934209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9" event={"ID":"9a83a035-1fa4-4f07-9bf4-31be8f6e5661","Type":"ContainerStarted","Data":"b98e3b178baaff114f22924cd125c2082f19581e6b4d8af31cbe029f9198a666"}
Jul 11 03:47:42.878876 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:42.878836    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"]
Jul 11 03:47:42.879255 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:42.879061    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" podUID="f6590914-5575-4fbc-aed1-b165b0833027" containerName="discovery" containerID="cri-o://7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b" gracePeriod=30
Jul 11 03:47:42.911034 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:42.911002    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-djcsm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.22:8080/ready\": dial tcp 10.132.0.22:8080: connect: connection refused" start-of-body=
Jul 11 03:47:42.911216 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:42.911067    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" podUID="f6590914-5575-4fbc-aed1-b165b0833027" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.22:8080/ready\": dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 11 03:47:43.500645 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.500619    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:43.566184 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566148    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-dns-cert\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566361 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-ca-configmap\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566361 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6590914-5575-4fbc-aed1-b165b0833027-local-certs\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566361 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566294    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-cacerts\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566361 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fskqm\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-kube-api-access-fskqm\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566583 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-kubeconfig\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566583 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566387    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-istio-token\") pod \"f6590914-5575-4fbc-aed1-b165b0833027\" (UID: \"f6590914-5575-4fbc-aed1-b165b0833027\") "
Jul 11 03:47:43.566677 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.566597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:47:43.569331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.569302    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:47:43.569331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.569313    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:47:43.569524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.569397    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:47:43.569677 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.569655    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6590914-5575-4fbc-aed1-b165b0833027-local-certs" (OuterVolumeSpecName: "local-certs") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 03:47:43.569798 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.569745    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-cacerts" (OuterVolumeSpecName: "cacerts") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:47:43.569798 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.569765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-kube-api-access-fskqm" (OuterVolumeSpecName: "kube-api-access-fskqm") pod "f6590914-5575-4fbc-aed1-b165b0833027" (UID: "f6590914-5575-4fbc-aed1-b165b0833027"). InnerVolumeSpecName "kube-api-access-fskqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:47:43.667951 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.667922    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-dns-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.667951 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.667951    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6590914-5575-4fbc-aed1-b165b0833027-istio-csr-ca-configmap\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.668206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.667965    2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6590914-5575-4fbc-aed1-b165b0833027-local-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.668206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.667978    2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-cacerts\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.668206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.667989    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskqm\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-kube-api-access-fskqm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.668206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.668002    2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6590914-5575-4fbc-aed1-b165b0833027-istio-kubeconfig\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.668206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.668013    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6590914-5575-4fbc-aed1-b165b0833027-istio-token\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:47:43.945605 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.945516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4" event={"ID":"ff10d8c5-3d3a-4935-a4f6-d193469e6e6c","Type":"ContainerStarted","Data":"2e0ce1ae9e251faf54eb44baaa572e6b302d6748e977173c0626273b1472d748"}
Jul 11 03:47:43.945770 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.945646    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:47:43.947042 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.947014    2582 generic.go:356] "Generic (PLEG): container finished" podID="f6590914-5575-4fbc-aed1-b165b0833027" containerID="7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b" exitCode=0
Jul 11 03:47:43.947206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.947071    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"
Jul 11 03:47:43.947206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.947123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" event={"ID":"f6590914-5575-4fbc-aed1-b165b0833027","Type":"ContainerDied","Data":"7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b"}
Jul 11 03:47:43.947206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.947182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-djcsm" event={"ID":"f6590914-5575-4fbc-aed1-b165b0833027","Type":"ContainerDied","Data":"374a7a564c417cbc4309731c841649bbe82c70b764f20d48c00fcc3ef26b253a"}
Jul 11 03:47:43.947206 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.947209    2582 scope.go:117] "RemoveContainer" containerID="7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b"
Jul 11 03:47:43.963264 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.963216    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4" podStartSLOduration=1.460438983 podStartE2EDuration="3.96320067s" totalImagesPullingTime="2.502761687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:43.962491038 +0000 UTC m=+308.831937602" watchObservedRunningTime="2026-07-11 03:47:43.96320067 +0000 UTC m=+308.832647235"
Jul 11 03:47:43.976762 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.976732    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"]
Jul 11 03:47:43.980784 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:43.980761    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-djcsm"]
Jul 11 03:47:44.183408 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.183371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:44.287599 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.287563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:44.418416 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.418393    2582 scope.go:117] "RemoveContainer" containerID="7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b"
Jul 11 03:47:44.418766 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:47:44.418739    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b\": container with ID starting with 7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b not found: ID does not exist" containerID="7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b"
Jul 11 03:47:44.418866 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.418774    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b"} err="failed to get container status \"7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b\": rpc error: code = NotFound desc = could not find container \"7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b\": container with ID starting with 7f061289a9349f3c617bab29d15b1fe156d9a3bc0885b13bf0bd67dbe63db14b not found: ID does not exist"
Jul 11 03:47:44.953698 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.953604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9" event={"ID":"9a83a035-1fa4-4f07-9bf4-31be8f6e5661","Type":"ContainerStarted","Data":"89a941a29d4f963fa0e5dcb59eeccf02b896aa0dd3d654184cae6a603f5c2df1"}
Jul 11 03:47:44.953698 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.953690    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:47:44.979116 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:44.979057    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9" podStartSLOduration=2.037912095 podStartE2EDuration="4.979040493s" totalImagesPullingTime="2.941128398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:44.978377232 +0000 UTC m=+309.847823797" watchObservedRunningTime="2026-07-11 03:47:44.979040493 +0000 UTC m=+309.848487056"
Jul 11 03:47:45.645269 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:45.645241    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6590914-5575-4fbc-aed1-b165b0833027" path="/var/lib/kubelet/pods/f6590914-5575-4fbc-aed1-b165b0833027/volumes"
Jul 11 03:47:46.177334 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:46.177268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:47:46.913002 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:47:46.912970    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4hfnt"
Jul 11 03:48:14.960630 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:14.960595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bffn4"
Jul 11 03:48:15.961538 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:15.961507    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vd8l9"
Jul 11 03:48:30.757735 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.757661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:48:30.865466 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.865433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:48:30.871733 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.871707    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rkksz"]
Jul 11 03:48:30.872008 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.871997    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6590914-5575-4fbc-aed1-b165b0833027" containerName="discovery"
Jul 11 03:48:30.872047 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.872010    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6590914-5575-4fbc-aed1-b165b0833027" containerName="discovery"
Jul 11 03:48:30.872080 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.872071    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6590914-5575-4fbc-aed1-b165b0833027" containerName="discovery"
Jul 11 03:48:30.874834 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.874816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:30.877754 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.877733    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 11 03:48:30.879609 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.878858    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 11 03:48:30.879915 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.878989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bxlrr\""
Jul 11 03:48:30.883710 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.883689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rkksz"]
Jul 11 03:48:30.925460 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.925431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:48:30.941477 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:30.941446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkh8\" (UniqueName: \"kubernetes.io/projected/8f06c488-1ba3-4b08-bf80-add86c17981e-kube-api-access-xgkh8\") pod \"kuadrant-operator-catalog-rkksz\" (UID: \"8f06c488-1ba3-4b08-bf80-add86c17981e\") " pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:31.042145 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.042045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkh8\" (UniqueName: \"kubernetes.io/projected/8f06c488-1ba3-4b08-bf80-add86c17981e-kube-api-access-xgkh8\") pod \"kuadrant-operator-catalog-rkksz\" (UID: \"8f06c488-1ba3-4b08-bf80-add86c17981e\") " pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:31.049838 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.049815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkh8\" (UniqueName: \"kubernetes.io/projected/8f06c488-1ba3-4b08-bf80-add86c17981e-kube-api-access-xgkh8\") pod \"kuadrant-operator-catalog-rkksz\" (UID: \"8f06c488-1ba3-4b08-bf80-add86c17981e\") " pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:31.188753 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.188712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:31.233624 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.233577    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rkksz"]
Jul 11 03:48:31.318489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.318415    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rkksz"]
Jul 11 03:48:31.320728 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:48:31.320699    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f06c488_1ba3_4b08_bf80_add86c17981e.slice/crio-b0f721144ad63128f080cf9bb38a8401e8f8125ce477bce67bc69303377e3f51 WatchSource:0}: Error finding container b0f721144ad63128f080cf9bb38a8401e8f8125ce477bce67bc69303377e3f51: Status 404 returned error can't find the container with id b0f721144ad63128f080cf9bb38a8401e8f8125ce477bce67bc69303377e3f51
Jul 11 03:48:31.444905 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.444865    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6ghz9"]
Jul 11 03:48:31.449244 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.449227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:31.455167 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.455141    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6ghz9"]
Jul 11 03:48:31.545911 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.545878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhlj\" (UniqueName: \"kubernetes.io/projected/69f17c55-8391-483d-9a85-8d8014b2eb87-kube-api-access-qjhlj\") pod \"kuadrant-operator-catalog-6ghz9\" (UID: \"69f17c55-8391-483d-9a85-8d8014b2eb87\") " pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:31.646536 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.646448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhlj\" (UniqueName: \"kubernetes.io/projected/69f17c55-8391-483d-9a85-8d8014b2eb87-kube-api-access-qjhlj\") pod \"kuadrant-operator-catalog-6ghz9\" (UID: \"69f17c55-8391-483d-9a85-8d8014b2eb87\") " pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:31.654601 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.654571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhlj\" (UniqueName: \"kubernetes.io/projected/69f17c55-8391-483d-9a85-8d8014b2eb87-kube-api-access-qjhlj\") pod \"kuadrant-operator-catalog-6ghz9\" (UID: \"69f17c55-8391-483d-9a85-8d8014b2eb87\") " pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:31.666224 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.666183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:48:31.759154 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.759122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:31.866807 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.866773    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:48:31.903642 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:31.903558    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6ghz9"]
Jul 11 03:48:31.933350 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:48:31.933322    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f17c55_8391_483d_9a85_8d8014b2eb87.slice/crio-d6ec34d273200f12a8644858464cddd7035b7cfeb6929f632be58dc150c7ceaa WatchSource:0}: Error finding container d6ec34d273200f12a8644858464cddd7035b7cfeb6929f632be58dc150c7ceaa: Status 404 returned error can't find the container with id d6ec34d273200f12a8644858464cddd7035b7cfeb6929f632be58dc150c7ceaa
Jul 11 03:48:32.145809 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:32.145770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9" event={"ID":"69f17c55-8391-483d-9a85-8d8014b2eb87","Type":"ContainerStarted","Data":"d6ec34d273200f12a8644858464cddd7035b7cfeb6929f632be58dc150c7ceaa"}
Jul 11 03:48:32.147021 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:32.146994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rkksz" event={"ID":"8f06c488-1ba3-4b08-bf80-add86c17981e","Type":"ContainerStarted","Data":"b0f721144ad63128f080cf9bb38a8401e8f8125ce477bce67bc69303377e3f51"}
Jul 11 03:48:34.157001 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.156968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9" event={"ID":"69f17c55-8391-483d-9a85-8d8014b2eb87","Type":"ContainerStarted","Data":"910f18ed1ea9fc7b1a0ce4def73c5e45a20ecc95c939a54c317851fd1b83247b"}
Jul 11 03:48:34.158404 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.158383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rkksz" event={"ID":"8f06c488-1ba3-4b08-bf80-add86c17981e","Type":"ContainerStarted","Data":"e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4"}
Jul 11 03:48:34.158512 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.158446    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rkksz" podUID="8f06c488-1ba3-4b08-bf80-add86c17981e" containerName="registry-server" containerID="cri-o://e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4" gracePeriod=2
Jul 11 03:48:34.172753 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.172711    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9" podStartSLOduration=1.958970641 podStartE2EDuration="3.172698258s" totalImagesPullingTime="1.213727617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:48:34.17100856 +0000 UTC m=+359.040455124" watchObservedRunningTime="2026-07-11 03:48:34.172698258 +0000 UTC m=+359.042144822"
Jul 11 03:48:34.187017 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.186974    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rkksz" podStartSLOduration=2.361303479 podStartE2EDuration="4.186959724s" totalImagesPullingTime="1.825656245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:48:34.185486322 +0000 UTC m=+359.054932887" watchObservedRunningTime="2026-07-11 03:48:34.186959724 +0000 UTC m=+359.056406287"
Jul 11 03:48:34.404087 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.404063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:34.470528 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.470445    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgkh8\" (UniqueName: \"kubernetes.io/projected/8f06c488-1ba3-4b08-bf80-add86c17981e-kube-api-access-xgkh8\") pod \"8f06c488-1ba3-4b08-bf80-add86c17981e\" (UID: \"8f06c488-1ba3-4b08-bf80-add86c17981e\") "
Jul 11 03:48:34.472734 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.472710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f06c488-1ba3-4b08-bf80-add86c17981e-kube-api-access-xgkh8" (OuterVolumeSpecName: "kube-api-access-xgkh8") pod "8f06c488-1ba3-4b08-bf80-add86c17981e" (UID: "8f06c488-1ba3-4b08-bf80-add86c17981e"). InnerVolumeSpecName "kube-api-access-xgkh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:48:34.571108 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:34.571073    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgkh8\" (UniqueName: \"kubernetes.io/projected/8f06c488-1ba3-4b08-bf80-add86c17981e-kube-api-access-xgkh8\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:48:35.165993 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.165952    2582 generic.go:356] "Generic (PLEG): container finished" podID="8f06c488-1ba3-4b08-bf80-add86c17981e" containerID="e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4" exitCode=0
Jul 11 03:48:35.166450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.166011    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rkksz"
Jul 11 03:48:35.166450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.166045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rkksz" event={"ID":"8f06c488-1ba3-4b08-bf80-add86c17981e","Type":"ContainerDied","Data":"e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4"}
Jul 11 03:48:35.166450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.166138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rkksz" event={"ID":"8f06c488-1ba3-4b08-bf80-add86c17981e","Type":"ContainerDied","Data":"b0f721144ad63128f080cf9bb38a8401e8f8125ce477bce67bc69303377e3f51"}
Jul 11 03:48:35.166450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.166160    2582 scope.go:117] "RemoveContainer" containerID="e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4"
Jul 11 03:48:35.174800 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.174780    2582 scope.go:117] "RemoveContainer" containerID="e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4"
Jul 11 03:48:35.175040 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:48:35.175022    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4\": container with ID starting with e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4 not found: ID does not exist" containerID="e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4"
Jul 11 03:48:35.175089 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.175049    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4"} err="failed to get container status \"e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4\": rpc error: code = NotFound desc = could not find container \"e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4\": container with ID starting with e492689335efd60d503254210375a3d7174caed0a4f2cf4af8a5332bf53ae5d4 not found: ID does not exist"
Jul 11 03:48:35.185522 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.185504    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rkksz"]
Jul 11 03:48:35.187979 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.187960    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rkksz"]
Jul 11 03:48:35.645113 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:35.645058    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f06c488-1ba3-4b08-bf80-add86c17981e" path="/var/lib/kubelet/pods/8f06c488-1ba3-4b08-bf80-add86c17981e/volumes"
Jul 11 03:48:41.759486 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:41.759449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:41.759945 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:41.759526    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:41.781552 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:41.781527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:42.217047 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:42.216976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6ghz9"
Jul 11 03:48:59.603073 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:48:59.603042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:03.452684 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:03.452642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:04.560200 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.560166    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"]
Jul 11 03:49:04.560566 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.560453    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f06c488-1ba3-4b08-bf80-add86c17981e" containerName="registry-server"
Jul 11 03:49:04.560566 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.560463    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f06c488-1ba3-4b08-bf80-add86c17981e" containerName="registry-server"
Jul 11 03:49:04.560566 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.560515    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f06c488-1ba3-4b08-bf80-add86c17981e" containerName="registry-server"
Jul 11 03:49:04.563672 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.563655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.566128 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.566088    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2468z\""
Jul 11 03:49:04.577727 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.577704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"]
Jul 11 03:49:04.705019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.704979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj4t\" (UniqueName: \"kubernetes.io/projected/b07dc936-4a75-4289-a6c4-fff7556d6f8f-kube-api-access-rtj4t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.705231 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.705023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b07dc936-4a75-4289-a6c4-fff7556d6f8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.805989 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.805947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj4t\" (UniqueName: \"kubernetes.io/projected/b07dc936-4a75-4289-a6c4-fff7556d6f8f-kube-api-access-rtj4t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.805989 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.805994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b07dc936-4a75-4289-a6c4-fff7556d6f8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.806373 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.806357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b07dc936-4a75-4289-a6c4-fff7556d6f8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.835776 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.835707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj4t\" (UniqueName: \"kubernetes.io/projected/b07dc936-4a75-4289-a6c4-fff7556d6f8f-kube-api-access-rtj4t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.873713 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.873686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:04.879958 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:04.879929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:05.033573 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:05.033466    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"]
Jul 11 03:49:05.035847 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:05.035818    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07dc936_4a75_4289_a6c4_fff7556d6f8f.slice/crio-5a4727b3c0bbba6709d942f44da90a8ac7327ace5699f4287fd4c2d43e6c0e55 WatchSource:0}: Error finding container 5a4727b3c0bbba6709d942f44da90a8ac7327ace5699f4287fd4c2d43e6c0e55: Status 404 returned error can't find the container with id 5a4727b3c0bbba6709d942f44da90a8ac7327ace5699f4287fd4c2d43e6c0e55
Jul 11 03:49:05.200682 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:05.200602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:05.288426 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:05.288394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl" event={"ID":"b07dc936-4a75-4289-a6c4-fff7556d6f8f","Type":"ContainerStarted","Data":"5a4727b3c0bbba6709d942f44da90a8ac7327ace5699f4287fd4c2d43e6c0e55"}
Jul 11 03:49:06.277299 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:06.277265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:07.109455 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.109409    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"]
Jul 11 03:49:07.116183 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.116162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:07.118646 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.118625    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-982js\""
Jul 11 03:49:07.121950 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.121925    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"]
Jul 11 03:49:07.225534 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.225501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzqs\" (UniqueName: \"kubernetes.io/projected/650de894-ccd5-4088-843e-0fbb21075709-kube-api-access-7pzqs\") pod \"authorino-operator-75cc8d5cb7-79jz4\" (UID: \"650de894-ccd5-4088-843e-0fbb21075709\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:07.326680 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.326636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzqs\" (UniqueName: \"kubernetes.io/projected/650de894-ccd5-4088-843e-0fbb21075709-kube-api-access-7pzqs\") pod \"authorino-operator-75cc8d5cb7-79jz4\" (UID: \"650de894-ccd5-4088-843e-0fbb21075709\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:07.336032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.336002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzqs\" (UniqueName: \"kubernetes.io/projected/650de894-ccd5-4088-843e-0fbb21075709-kube-api-access-7pzqs\") pod \"authorino-operator-75cc8d5cb7-79jz4\" (UID: \"650de894-ccd5-4088-843e-0fbb21075709\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:07.429977 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.429881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:07.575187 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.575151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"]
Jul 11 03:49:07.578268 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:07.578238    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650de894_ccd5_4088_843e_0fbb21075709.slice/crio-a1215f51aea80cf99864ec2bd9367ff0681debcfc7d5454fe4af2385a5f74d2d WatchSource:0}: Error finding container a1215f51aea80cf99864ec2bd9367ff0681debcfc7d5454fe4af2385a5f74d2d: Status 404 returned error can't find the container with id a1215f51aea80cf99864ec2bd9367ff0681debcfc7d5454fe4af2385a5f74d2d
Jul 11 03:49:07.828974 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.828944    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:07.922282 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:07.922245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:08.304339 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:08.304262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4" event={"ID":"650de894-ccd5-4088-843e-0fbb21075709","Type":"ContainerStarted","Data":"a1215f51aea80cf99864ec2bd9367ff0681debcfc7d5454fe4af2385a5f74d2d"}
Jul 11 03:49:09.402937 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.402903    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"]
Jul 11 03:49:09.406359 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.406332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:09.409690 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.409668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xfdjd\""
Jul 11 03:49:09.410058 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.410020    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 11 03:49:09.418327 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.418302    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"]
Jul 11 03:49:09.548267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.548223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64qd\" (UniqueName: \"kubernetes.io/projected/315db466-92b7-423c-99ef-a6f2c0459199-kube-api-access-l64qd\") pod \"dns-operator-controller-manager-864dd78c4c-kswfr\" (UID: \"315db466-92b7-423c-99ef-a6f2c0459199\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:09.649157 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.649123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64qd\" (UniqueName: \"kubernetes.io/projected/315db466-92b7-423c-99ef-a6f2c0459199-kube-api-access-l64qd\") pod \"dns-operator-controller-manager-864dd78c4c-kswfr\" (UID: \"315db466-92b7-423c-99ef-a6f2c0459199\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:09.660031 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.659961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64qd\" (UniqueName: \"kubernetes.io/projected/315db466-92b7-423c-99ef-a6f2c0459199-kube-api-access-l64qd\") pod \"dns-operator-controller-manager-864dd78c4c-kswfr\" (UID: \"315db466-92b7-423c-99ef-a6f2c0459199\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:09.718606 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:09.718568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:10.024005 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:10.023969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:11.083277 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.083256    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"]
Jul 11 03:49:11.086067 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:11.086037    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315db466_92b7_423c_99ef_a6f2c0459199.slice/crio-5c9d9ea06838ec2bf7dec1d10fb71819f5fe8f9ac2cd1859550442c0a41a59bd WatchSource:0}: Error finding container 5c9d9ea06838ec2bf7dec1d10fb71819f5fe8f9ac2cd1859550442c0a41a59bd: Status 404 returned error can't find the container with id 5c9d9ea06838ec2bf7dec1d10fb71819f5fe8f9ac2cd1859550442c0a41a59bd
Jul 11 03:49:11.320283 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.320245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl" event={"ID":"b07dc936-4a75-4289-a6c4-fff7556d6f8f","Type":"ContainerStarted","Data":"c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9"}
Jul 11 03:49:11.320463 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.320353    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:11.321513 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.321486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr" event={"ID":"315db466-92b7-423c-99ef-a6f2c0459199","Type":"ContainerStarted","Data":"5c9d9ea06838ec2bf7dec1d10fb71819f5fe8f9ac2cd1859550442c0a41a59bd"}
Jul 11 03:49:11.322828 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.322810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4" event={"ID":"650de894-ccd5-4088-843e-0fbb21075709","Type":"ContainerStarted","Data":"8364a46b7b858731c30bdccb6072942575cf56e626d56df7fc486eecf5f2d047"}
Jul 11 03:49:11.322939 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.322916    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:11.339340 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.339298    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl" podStartSLOduration=1.4230150799999999 podStartE2EDuration="7.339283157s" totalImagesPullingTime="5.916268077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:11.336923437 +0000 UTC m=+396.206370012" watchObservedRunningTime="2026-07-11 03:49:11.339283157 +0000 UTC m=+396.208729720"
Jul 11 03:49:11.358338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.358285    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4" podStartSLOduration=0.943012077 podStartE2EDuration="4.358269493s" totalImagesPullingTime="3.415257416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:11.356994626 +0000 UTC m=+396.226441182" watchObservedRunningTime="2026-07-11 03:49:11.358269493 +0000 UTC m=+396.227716128"
Jul 11 03:49:11.403003 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.402979    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"]
Jul 11 03:49:11.411987 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.411960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:11.414672 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.414641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-k8k6d\""
Jul 11 03:49:11.415028 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.415010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"]
Jul 11 03:49:11.563763 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.563726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncz5\" (UniqueName: \"kubernetes.io/projected/52ccbf66-8694-4c49-a492-8943ee8027b6-kube-api-access-6ncz5\") pod \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" (UID: \"52ccbf66-8694-4c49-a492-8943ee8027b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:11.664581 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.664509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncz5\" (UniqueName: \"kubernetes.io/projected/52ccbf66-8694-4c49-a492-8943ee8027b6-kube-api-access-6ncz5\") pod \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" (UID: \"52ccbf66-8694-4c49-a492-8943ee8027b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:11.673428 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.673407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncz5\" (UniqueName: \"kubernetes.io/projected/52ccbf66-8694-4c49-a492-8943ee8027b6-kube-api-access-6ncz5\") pod \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" (UID: \"52ccbf66-8694-4c49-a492-8943ee8027b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:11.723088 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.723049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:11.833697 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:11.833661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:12.055084 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:12.055002    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"]
Jul 11 03:49:12.122215 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:12.122182    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ccbf66_8694_4c49_a492_8943ee8027b6.slice/crio-4e1aa7fb66bc28b23f53959358e547d44a118c7fb34343713aa14633dc09c4d1 WatchSource:0}: Error finding container 4e1aa7fb66bc28b23f53959358e547d44a118c7fb34343713aa14633dc09c4d1: Status 404 returned error can't find the container with id 4e1aa7fb66bc28b23f53959358e547d44a118c7fb34343713aa14633dc09c4d1
Jul 11 03:49:12.330731 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:12.330695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8" event={"ID":"52ccbf66-8694-4c49-a492-8943ee8027b6","Type":"ContainerStarted","Data":"4e1aa7fb66bc28b23f53959358e547d44a118c7fb34343713aa14633dc09c4d1"}
Jul 11 03:49:13.605685 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.605572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:13.706380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.706342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:13.992668 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.992634    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"]
Jul 11 03:49:13.996332 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.996299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:13.997334 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.997303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:13.998826 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.998803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 11 03:49:13.998939 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.998813    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p7m6x\""
Jul 11 03:49:13.998988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:13.998964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 11 03:49:14.003558 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.003532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"]
Jul 11 03:49:14.087782 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.087755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1de9211-8287-44cf-9608-a949ba0ad8b4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.087954 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.087801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1de9211-8287-44cf-9608-a949ba0ad8b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.087954 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.087868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hjl\" (UniqueName: \"kubernetes.io/projected/e1de9211-8287-44cf-9608-a949ba0ad8b4-kube-api-access-g4hjl\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.188700 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.188674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hjl\" (UniqueName: \"kubernetes.io/projected/e1de9211-8287-44cf-9608-a949ba0ad8b4-kube-api-access-g4hjl\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.188794 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.188728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1de9211-8287-44cf-9608-a949ba0ad8b4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.188794 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.188761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1de9211-8287-44cf-9608-a949ba0ad8b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.189403 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.189386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1de9211-8287-44cf-9608-a949ba0ad8b4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.191019 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.191002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1de9211-8287-44cf-9608-a949ba0ad8b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.199028 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.199003    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:14.201266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.201250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hjl\" (UniqueName: \"kubernetes.io/projected/e1de9211-8287-44cf-9608-a949ba0ad8b4-kube-api-access-g4hjl\") pod \"kuadrant-console-plugin-66b5774985-7x6sp\" (UID: \"e1de9211-8287-44cf-9608-a949ba0ad8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.308622 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.308536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"
Jul 11 03:49:14.342410 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.342381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr" event={"ID":"315db466-92b7-423c-99ef-a6f2c0459199","Type":"ContainerStarted","Data":"e5dcd7fd9d67e1a27b18ccf99aa7d9ccf720caae0d670cefa6b9e49fd9b0e0b7"}
Jul 11 03:49:14.342596 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.342451    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:14.344060 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.344039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8" event={"ID":"52ccbf66-8694-4c49-a492-8943ee8027b6","Type":"ContainerStarted","Data":"206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734"}
Jul 11 03:49:14.344210 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.344151    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:14.358262 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.358215    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr" podStartSLOduration=3.29284691 podStartE2EDuration="5.358198678s" totalImagesPullingTime="2.065351768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:14.357512143 +0000 UTC m=+399.226958711" watchObservedRunningTime="2026-07-11 03:49:14.358198678 +0000 UTC m=+399.227645245"
Jul 11 03:49:14.374500 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.374452    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8" podStartSLOduration=1.393397477 podStartE2EDuration="3.374421545s" totalImagesPullingTime="1.981024068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:14.373389816 +0000 UTC m=+399.242836380" watchObservedRunningTime="2026-07-11 03:49:14.374421545 +0000 UTC m=+399.243868108"
Jul 11 03:49:14.442563 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:14.442540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp"]
Jul 11 03:49:14.444713 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:14.444688    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1de9211_8287_44cf_9608_a949ba0ad8b4.slice/crio-498c01f6abbd4627425de13cd47cb9e9dd1119d93e11c2d3f1bafb4b09c15c93 WatchSource:0}: Error finding container 498c01f6abbd4627425de13cd47cb9e9dd1119d93e11c2d3f1bafb4b09c15c93: Status 404 returned error can't find the container with id 498c01f6abbd4627425de13cd47cb9e9dd1119d93e11c2d3f1bafb4b09c15c93
Jul 11 03:49:15.351895 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:15.351854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp" event={"ID":"e1de9211-8287-44cf-9608-a949ba0ad8b4","Type":"ContainerStarted","Data":"498c01f6abbd4627425de13cd47cb9e9dd1119d93e11c2d3f1bafb4b09c15c93"}
Jul 11 03:49:22.333518 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:22.333485    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:22.334090 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:22.333635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-79jz4"
Jul 11 03:49:22.877170 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:22.877131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:22.983920 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:22.983869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:23.988350 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:23.988318    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"]
Jul 11 03:49:23.998749 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:23.998714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.005816 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.005789    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"]
Jul 11 03:49:24.087077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.087046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"]
Jul 11 03:49:24.087373 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.087315    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" containerName="manager" containerID="cri-o://c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9" gracePeriod=2
Jul 11 03:49:24.089727 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.089686    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:24.090988 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.090936    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"]
Jul 11 03:49:24.104570 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.104549    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"]
Jul 11 03:49:24.105034 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.105006    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-c85qz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306"
Jul 11 03:49:24.110792 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.110746    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"]
Jul 11 03:49:24.113048 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.113025    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"]
Jul 11 03:49:24.113764 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.113740    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" containerName="manager"
Jul 11 03:49:24.113863 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.113768    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" containerName="manager"
Jul 11 03:49:24.113863 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.113856    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" containerName="manager"
Jul 11 03:49:24.117450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.117421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.119346 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.119324    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"]
Jul 11 03:49:24.119442 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.119335    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:24.119555 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.119532    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" containerName="manager" containerID="cri-o://206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734" gracePeriod=2
Jul 11 03:49:24.121392 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.121374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:24.142087 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.142018    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"]
Jul 11 03:49:24.144158 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.144090    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"]
Jul 11 03:49:24.147683 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.147660    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"]
Jul 11 03:49:24.147788 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.147757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.157295 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.157257    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"]
Jul 11 03:49:24.175526 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.175411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85qz\" (UniqueName: \"kubernetes.io/projected/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-kube-api-access-c85qz\") pod \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" (UID: \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.175526 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.175474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" (UID: \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.178451 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.178417    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:24.180865 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.180833    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:24.276269 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85qz\" (UniqueName: \"kubernetes.io/projected/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-kube-api-access-c85qz\") pod \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" (UID: \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.276269 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" (UID: \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.276470 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8j4\" (UniqueName: \"kubernetes.io/projected/c62c474b-fc0f-4db3-a584-a8b6a705fb65-kube-api-access-lt8j4\") pod \"kuadrant-operator-controller-manager-86878c769f-mdcwt\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.276470 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5d22b98-c804-40d5-8331-04a650e0063f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8nw6s\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.276470 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbgj\" (UniqueName: \"kubernetes.io/projected/e5d22b98-c804-40d5-8331-04a650e0063f-kube-api-access-2cbgj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8nw6s\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.276470 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c62c474b-fc0f-4db3-a584-a8b6a705fb65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mdcwt\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.276696 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.276674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" (UID: \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.281417 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.281393    2582 projected.go:196] Error preparing data for projected volume kube-api-access-c85qz for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-dvp6v" not found
Jul 11 03:49:24.281582 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.281488    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-kube-api-access-c85qz podName:9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306 nodeName:}" failed. No retries permitted until 2026-07-11 03:49:24.781464887 +0000 UTC m=+409.650911443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c85qz" (UniqueName: "kubernetes.io/projected/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-kube-api-access-c85qz") pod "kuadrant-operator-controller-manager-86878c769f-dvp6v" (UID: "9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-dvp6v" not found
Jul 11 03:49:24.377566 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.377529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8j4\" (UniqueName: \"kubernetes.io/projected/c62c474b-fc0f-4db3-a584-a8b6a705fb65-kube-api-access-lt8j4\") pod \"kuadrant-operator-controller-manager-86878c769f-mdcwt\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.377566 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.377569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5d22b98-c804-40d5-8331-04a650e0063f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8nw6s\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.377822 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.377594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbgj\" (UniqueName: \"kubernetes.io/projected/e5d22b98-c804-40d5-8331-04a650e0063f-kube-api-access-2cbgj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8nw6s\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.377822 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.377634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c62c474b-fc0f-4db3-a584-a8b6a705fb65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mdcwt\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.378036 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.378005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5d22b98-c804-40d5-8331-04a650e0063f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8nw6s\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.378179 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.378084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c62c474b-fc0f-4db3-a584-a8b6a705fb65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mdcwt\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.394054 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.394025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8j4\" (UniqueName: \"kubernetes.io/projected/c62c474b-fc0f-4db3-a584-a8b6a705fb65-kube-api-access-lt8j4\") pod \"kuadrant-operator-controller-manager-86878c769f-mdcwt\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.395492 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.395470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.398252 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.398203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbgj\" (UniqueName: \"kubernetes.io/projected/e5d22b98-c804-40d5-8331-04a650e0063f-kube-api-access-2cbgj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8nw6s\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.398361 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.398262    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:24.400576 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.400547    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:24.402792 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.402766    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.420575 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.420549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.423204 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.423174    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:24.425328 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.425306    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:24.427563 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:24.427531    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:24.500588 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.500552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:24.506652 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.506610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:24.579787 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.579298    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-extensions-socket-volume\") pod \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\" (UID: \"9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306\") "
Jul 11 03:49:24.579787 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.579524    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85qz\" (UniqueName: \"kubernetes.io/projected/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-kube-api-access-c85qz\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:24.580069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.580040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" (UID: "9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 03:49:24.680990 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:24.680950    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306-extensions-socket-volume\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:25.284828 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:25.284796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:25.355174 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:25.355147    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kswfr"
Jul 11 03:49:25.357544 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.357496    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:25.359515 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.359479    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:25.361390 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.361359    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:25.385816 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:25.385779    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:25.401520 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:25.401490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:25.404207 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.404168    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:25.406167 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.406132    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:25.408064 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.408033    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:25.412308 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.412277    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:25.414165 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.414131    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:25.415941 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.415910    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:25.644569 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.644471    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:25.646365 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.646321    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dvp6v\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dvp6v"
Jul 11 03:49:25.647409 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:25.647339    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306" path="/var/lib/kubelet/pods/9304b9bc-0e46-4cfd-ab5f-e5a8c21d5306/volumes"
Jul 11 03:49:25.648146 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:25.648122    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:30.019720 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:30.019636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:30.121264 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:30.121227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:33.822741 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:33.822701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:33.887160 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:33.887038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:37.199201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.199170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.202537 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.201690    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.203601 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.203573    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"]
Jul 11 03:49:37.204907 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:37.204883    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62c474b_fc0f_4db3_a584_a8b6a705fb65.slice/crio-ca8b59746419f1c0f8f542dd29b95a0384625518d2cb2bf908f14ca8be92e779 WatchSource:0}: Error finding container ca8b59746419f1c0f8f542dd29b95a0384625518d2cb2bf908f14ca8be92e779: Status 404 returned error can't find the container with id ca8b59746419f1c0f8f542dd29b95a0384625518d2cb2bf908f14ca8be92e779
Jul 11 03:49:37.209272 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.209254    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:37.211070 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.211047    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:37.212821 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.212792    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.216574 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.216553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"]
Jul 11 03:49:37.219734 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:37.219706    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d22b98_c804_40d5_8331_04a650e0063f.slice/crio-86743efb79c1f48a039fc42f7937d16b25e4a9e2a1347ca6471c08e7d08fa974 WatchSource:0}: Error finding container 86743efb79c1f48a039fc42f7937d16b25e4a9e2a1347ca6471c08e7d08fa974: Status 404 returned error can't find the container with id 86743efb79c1f48a039fc42f7937d16b25e4a9e2a1347ca6471c08e7d08fa974
Jul 11 03:49:37.291514 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.291442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtj4t\" (UniqueName: \"kubernetes.io/projected/b07dc936-4a75-4289-a6c4-fff7556d6f8f-kube-api-access-rtj4t\") pod \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") "
Jul 11 03:49:37.291514 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.291511    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b07dc936-4a75-4289-a6c4-fff7556d6f8f-extensions-socket-volume\") pod \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\" (UID: \"b07dc936-4a75-4289-a6c4-fff7556d6f8f\") "
Jul 11 03:49:37.291688 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.291544    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ncz5\" (UniqueName: \"kubernetes.io/projected/52ccbf66-8694-4c49-a492-8943ee8027b6-kube-api-access-6ncz5\") pod \"52ccbf66-8694-4c49-a492-8943ee8027b6\" (UID: \"52ccbf66-8694-4c49-a492-8943ee8027b6\") "
Jul 11 03:49:37.292035 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.292001    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07dc936-4a75-4289-a6c4-fff7556d6f8f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b07dc936-4a75-4289-a6c4-fff7556d6f8f" (UID: "b07dc936-4a75-4289-a6c4-fff7556d6f8f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 03:49:37.294119 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.294074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ccbf66-8694-4c49-a492-8943ee8027b6-kube-api-access-6ncz5" (OuterVolumeSpecName: "kube-api-access-6ncz5") pod "52ccbf66-8694-4c49-a492-8943ee8027b6" (UID: "52ccbf66-8694-4c49-a492-8943ee8027b6"). InnerVolumeSpecName "kube-api-access-6ncz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:49:37.294194 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.294121    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07dc936-4a75-4289-a6c4-fff7556d6f8f-kube-api-access-rtj4t" (OuterVolumeSpecName: "kube-api-access-rtj4t") pod "b07dc936-4a75-4289-a6c4-fff7556d6f8f" (UID: "b07dc936-4a75-4289-a6c4-fff7556d6f8f"). InnerVolumeSpecName "kube-api-access-rtj4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:49:37.393054 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.393019    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b07dc936-4a75-4289-a6c4-fff7556d6f8f-extensions-socket-volume\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:37.393054 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.393048    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncz5\" (UniqueName: \"kubernetes.io/projected/52ccbf66-8694-4c49-a492-8943ee8027b6-kube-api-access-6ncz5\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:37.393262 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.393064    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtj4t\" (UniqueName: \"kubernetes.io/projected/b07dc936-4a75-4289-a6c4-fff7556d6f8f-kube-api-access-rtj4t\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:37.461054 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.461021    2582 generic.go:356] "Generic (PLEG): container finished" podID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" containerID="c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9" exitCode=0
Jul 11 03:49:37.461249 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.461067    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.461249 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.461130    2582 scope.go:117] "RemoveContainer" containerID="c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9"
Jul 11 03:49:37.463178 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.463141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s" event={"ID":"e5d22b98-c804-40d5-8331-04a650e0063f","Type":"ContainerStarted","Data":"464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272"}
Jul 11 03:49:37.463305 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.463199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s" event={"ID":"e5d22b98-c804-40d5-8331-04a650e0063f","Type":"ContainerStarted","Data":"86743efb79c1f48a039fc42f7937d16b25e4a9e2a1347ca6471c08e7d08fa974"}
Jul 11 03:49:37.463437 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.463358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:37.463510 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.463493    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.465422 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.465385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt" event={"ID":"c62c474b-fc0f-4db3-a584-a8b6a705fb65","Type":"ContainerStarted","Data":"72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01"}
Jul 11 03:49:37.465532 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.465436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt" event={"ID":"c62c474b-fc0f-4db3-a584-a8b6a705fb65","Type":"ContainerStarted","Data":"ca8b59746419f1c0f8f542dd29b95a0384625518d2cb2bf908f14ca8be92e779"}
Jul 11 03:49:37.465532 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.465433    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:37.465532 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.465476    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:37.467437 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.467404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp" event={"ID":"e1de9211-8287-44cf-9608-a949ba0ad8b4","Type":"ContainerStarted","Data":"cfdc57ce763990904c21b22d80932f8040da00f6f9711f5a77e4bd3c63978168"}
Jul 11 03:49:37.468904 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.468879    2582 generic.go:356] "Generic (PLEG): container finished" podID="52ccbf66-8694-4c49-a492-8943ee8027b6" containerID="206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734" exitCode=0
Jul 11 03:49:37.469012 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.468921    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:37.472812 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.472771    2582 scope.go:117] "RemoveContainer" containerID="c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9"
Jul 11 03:49:37.473093 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.473075    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9\": container with ID starting with c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9 not found: ID does not exist" containerID="c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9"
Jul 11 03:49:37.473201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.473119    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9"} err="failed to get container status \"c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9\": rpc error: code = NotFound desc = could not find container \"c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9\": container with ID starting with c4e77b2a9572d7f75ae23d78916b6e7c258e1cde656e75ab5678fd25d3342cf9 not found: ID does not exist"
Jul 11 03:49:37.473201 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.473142    2582 scope.go:117] "RemoveContainer" containerID="206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734"
Jul 11 03:49:37.481422 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.481405    2582 scope.go:117] "RemoveContainer" containerID="206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734"
Jul 11 03:49:37.481672 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.481658    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734\": container with ID starting with 206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734 not found: ID does not exist" containerID="206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734"
Jul 11 03:49:37.481708 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.481678    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734"} err="failed to get container status \"206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734\": rpc error: code = NotFound desc = could not find container \"206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734\": container with ID starting with 206d6d10174a5ffd730ba84cf8f6f8b3e5c62cf4ed5231acc61690b103a8b734 not found: ID does not exist"
Jul 11 03:49:37.486806 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.486768    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s" podStartSLOduration=13.486755198000001 podStartE2EDuration="13.486755198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:37.484916751 +0000 UTC m=+422.354363315" watchObservedRunningTime="2026-07-11 03:49:37.486755198 +0000 UTC m=+422.356201762"
Jul 11 03:49:37.501746 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.501696    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt" podStartSLOduration=13.501683958 podStartE2EDuration="13.501683958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:37.501475002 +0000 UTC m=+422.370921565" watchObservedRunningTime="2026-07-11 03:49:37.501683958 +0000 UTC m=+422.371130519"
Jul 11 03:49:37.503364 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.503340    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.505193 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.505171    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:37.519887 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.519853    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6sp" podStartSLOduration=1.8265049150000001 podStartE2EDuration="24.519841024s" totalImagesPullingTime="22.693336109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:37.51930222 +0000 UTC m=+422.388748776" watchObservedRunningTime="2026-07-11 03:49:37.519841024 +0000 UTC m=+422.389287579"
Jul 11 03:49:37.521112 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.521068    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-456fl\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-456fl"
Jul 11 03:49:37.522992 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:37.522969    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jtxz8\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jtxz8"
Jul 11 03:49:37.650656 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.650554    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" path="/var/lib/kubelet/pods/52ccbf66-8694-4c49-a492-8943ee8027b6/volumes"
Jul 11 03:49:37.651077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:37.651057    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b07dc936-4a75-4289-a6c4-fff7556d6f8f" path="/var/lib/kubelet/pods/b07dc936-4a75-4289-a6c4-fff7556d6f8f/volumes"
Jul 11 03:49:42.541504 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:42.541466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:42.640517 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:42.640470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:44.978812 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:44.978766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:45.048258 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:45.048222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:46.243832 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.243798    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f7cc5864-dl6b5"]
Jul 11 03:49:46.244245 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.244197    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" containerName="manager"
Jul 11 03:49:46.244245 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.244210    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" containerName="manager"
Jul 11 03:49:46.244331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.244283    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ccbf66-8694-4c49-a492-8943ee8027b6" containerName="manager"
Jul 11 03:49:46.302320 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.302277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:46.304364 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.304329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f7cc5864-dl6b5"]
Jul 11 03:49:46.304476 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.304414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.307356 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.307334    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 11 03:49:46.308358 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308338    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 11 03:49:46.308459 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308399    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 11 03:49:46.308574 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308560    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 11 03:49:46.308636 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308572    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 11 03:49:46.308756 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6nww6\""
Jul 11 03:49:46.308876 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 11 03:49:46.308986 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.308972    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 11 03:49:46.312522 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.312503    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 11 03:49:46.357259 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.357220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:46.367396 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-oauth-serving-cert\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.367476 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-console-config\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.367476 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-service-ca\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.367561 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-trusted-ca-bundle\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.367561 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5qv\" (UniqueName: \"kubernetes.io/projected/4a3447dc-15e1-402d-9926-156ab9c304c1-kube-api-access-6d5qv\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.367561 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3447dc-15e1-402d-9926-156ab9c304c1-console-oauth-config\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.367683 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.367590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3447dc-15e1-402d-9926-156ab9c304c1-console-serving-cert\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.436222 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.436178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:46.468371 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-console-config\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.468524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-service-ca\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.468524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-trusted-ca-bundle\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.468524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5qv\" (UniqueName: \"kubernetes.io/projected/4a3447dc-15e1-402d-9926-156ab9c304c1-kube-api-access-6d5qv\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.468524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3447dc-15e1-402d-9926-156ab9c304c1-console-oauth-config\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.468524 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3447dc-15e1-402d-9926-156ab9c304c1-console-serving-cert\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.468761 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.468722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-oauth-serving-cert\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.469373 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.469210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-console-config\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.469373 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.469286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-service-ca\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.469373 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.469307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-trusted-ca-bundle\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.469608 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.469394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3447dc-15e1-402d-9926-156ab9c304c1-oauth-serving-cert\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.471144 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.471091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3447dc-15e1-402d-9926-156ab9c304c1-console-oauth-config\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.471591 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.471570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3447dc-15e1-402d-9926-156ab9c304c1-console-serving-cert\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.475671 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.475649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5qv\" (UniqueName: \"kubernetes.io/projected/4a3447dc-15e1-402d-9926-156ab9c304c1-kube-api-access-6d5qv\") pod \"console-68f7cc5864-dl6b5\" (UID: \"4a3447dc-15e1-402d-9926-156ab9c304c1\") " pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.537227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.537151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:46.613755 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.613729    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:46.933950 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:46.933916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f7cc5864-dl6b5"]
Jul 11 03:49:46.937984 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:49:46.937959    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3447dc_15e1_402d_9926_156ab9c304c1.slice/crio-77786ea1e09d749ddfb621441e53df7e90ef15ed0820dbf78c5fad418fb8ed30 WatchSource:0}: Error finding container 77786ea1e09d749ddfb621441e53df7e90ef15ed0820dbf78c5fad418fb8ed30: Status 404 returned error can't find the container with id 77786ea1e09d749ddfb621441e53df7e90ef15ed0820dbf78c5fad418fb8ed30
Jul 11 03:49:47.040880 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:47.040851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:47.522596 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:47.522559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f7cc5864-dl6b5" event={"ID":"4a3447dc-15e1-402d-9926-156ab9c304c1","Type":"ContainerStarted","Data":"1993ca34e06bd2275a7cd06d73acba6add178b25c2fa8522599b83a34e561ec0"}
Jul 11 03:49:47.522986 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:47.522613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f7cc5864-dl6b5" event={"ID":"4a3447dc-15e1-402d-9926-156ab9c304c1","Type":"ContainerStarted","Data":"77786ea1e09d749ddfb621441e53df7e90ef15ed0820dbf78c5fad418fb8ed30"}
Jul 11 03:49:47.540719 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:47.540671    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f7cc5864-dl6b5" podStartSLOduration=1.540654699 podStartE2EDuration="1.540654699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:49:47.539439319 +0000 UTC m=+432.408885885" watchObservedRunningTime="2026-07-11 03:49:47.540654699 +0000 UTC m=+432.410101262"
Jul 11 03:49:48.478238 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:48.478197    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:49:48.478419 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:48.478390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:48.554043 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:48.554012    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"]
Jul 11 03:49:48.554477 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:48.554222    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s" podUID="e5d22b98-c804-40d5-8331-04a650e0063f" containerName="manager" containerID="cri-o://464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272" gracePeriod=10
Jul 11 03:49:49.316929 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.316901    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:49.392557 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.392527    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cbgj\" (UniqueName: \"kubernetes.io/projected/e5d22b98-c804-40d5-8331-04a650e0063f-kube-api-access-2cbgj\") pod \"e5d22b98-c804-40d5-8331-04a650e0063f\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") "
Jul 11 03:49:49.392710 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.392582    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5d22b98-c804-40d5-8331-04a650e0063f-extensions-socket-volume\") pod \"e5d22b98-c804-40d5-8331-04a650e0063f\" (UID: \"e5d22b98-c804-40d5-8331-04a650e0063f\") "
Jul 11 03:49:49.392955 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.392932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d22b98-c804-40d5-8331-04a650e0063f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e5d22b98-c804-40d5-8331-04a650e0063f" (UID: "e5d22b98-c804-40d5-8331-04a650e0063f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 03:49:49.394627 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.394600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d22b98-c804-40d5-8331-04a650e0063f-kube-api-access-2cbgj" (OuterVolumeSpecName: "kube-api-access-2cbgj") pod "e5d22b98-c804-40d5-8331-04a650e0063f" (UID: "e5d22b98-c804-40d5-8331-04a650e0063f"). InnerVolumeSpecName "kube-api-access-2cbgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:49:49.493508 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.493470    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbgj\" (UniqueName: \"kubernetes.io/projected/e5d22b98-c804-40d5-8331-04a650e0063f-kube-api-access-2cbgj\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:49.493508 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.493503    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5d22b98-c804-40d5-8331-04a650e0063f-extensions-socket-volume\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:49:49.533812 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.533784    2582 generic.go:356] "Generic (PLEG): container finished" podID="e5d22b98-c804-40d5-8331-04a650e0063f" containerID="464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272" exitCode=0
Jul 11 03:49:49.533972 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.533830    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"
Jul 11 03:49:49.533972 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.533870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s" event={"ID":"e5d22b98-c804-40d5-8331-04a650e0063f","Type":"ContainerDied","Data":"464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272"}
Jul 11 03:49:49.533972 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.533929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s" event={"ID":"e5d22b98-c804-40d5-8331-04a650e0063f","Type":"ContainerDied","Data":"86743efb79c1f48a039fc42f7937d16b25e4a9e2a1347ca6471c08e7d08fa974"}
Jul 11 03:49:49.533972 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.533960    2582 scope.go:117] "RemoveContainer" containerID="464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272"
Jul 11 03:49:49.544257 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.544241    2582 scope.go:117] "RemoveContainer" containerID="464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272"
Jul 11 03:49:49.544566 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:49:49.544544    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272\": container with ID starting with 464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272 not found: ID does not exist" containerID="464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272"
Jul 11 03:49:49.544638 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.544573    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272"} err="failed to get container status \"464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272\": rpc error: code = NotFound desc = could not find container \"464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272\": container with ID starting with 464499d22e17abda64c1c8a5ec3760263d663879d4b063e38aadfbda3f75c272 not found: ID does not exist"
Jul 11 03:49:49.555404 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.555373    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"]
Jul 11 03:49:49.559324 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.559302    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8nw6s"]
Jul 11 03:49:49.645541 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.645514    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d22b98-c804-40d5-8331-04a650e0063f" path="/var/lib/kubelet/pods/e5d22b98-c804-40d5-8331-04a650e0063f/volumes"
Jul 11 03:49:49.748306 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.748233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:49.854857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:49.854827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:53.744902 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:53.744864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:53.859430 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:53.859395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:56.614797 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:56.614706    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:56.614797 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:56.614780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:56.619634 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:56.619613    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:57.581268 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:57.581242    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f7cc5864-dl6b5"
Jul 11 03:49:57.722363 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:57.722333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:57.917380 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:57.917294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:49:57.968670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:49:57.968645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:03.942509 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:03.942477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:04.046632 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.046607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:04.345697 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.345665    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:04.471034 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.471001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:04.865196 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.865168    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"]
Jul 11 03:50:04.865509 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.865498    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d22b98-c804-40d5-8331-04a650e0063f" containerName="manager"
Jul 11 03:50:04.865554 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.865511    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d22b98-c804-40d5-8331-04a650e0063f" containerName="manager"
Jul 11 03:50:04.865590 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.865565    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d22b98-c804-40d5-8331-04a650e0063f" containerName="manager"
Jul 11 03:50:04.867438 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.867417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.869666 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.869649    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 11 03:50:04.869912 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.869896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vsv4c\""
Jul 11 03:50:04.878764 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.878740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"]
Jul 11 03:50:04.895613 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.895578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:04.926815 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.926781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc975611-2bb1-481c-b786-1399a8f95041-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.926815 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.926816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927038 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.926838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc975611-2bb1-481c-b786-1399a8f95041-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927038 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.926899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995bb\" (UniqueName: \"kubernetes.io/projected/dc975611-2bb1-481c-b786-1399a8f95041-kube-api-access-995bb\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927038 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.926941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc975611-2bb1-481c-b786-1399a8f95041-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927038 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.926978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927038 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.927014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927038 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.927037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.927276 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.927058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:04.998336 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:04.998300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:05.028318 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028505 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028505 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc975611-2bb1-481c-b786-1399a8f95041-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028505 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028505 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc975611-2bb1-481c-b786-1399a8f95041-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028505 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995bb\" (UniqueName: \"kubernetes.io/projected/dc975611-2bb1-481c-b786-1399a8f95041-kube-api-access-995bb\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028764 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc975611-2bb1-481c-b786-1399a8f95041-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028764 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028764 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.028906 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.029002 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.028974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.029118 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.029011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.029208 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.029073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.029366 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.029348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc975611-2bb1-481c-b786-1399a8f95041-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.030814 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.030792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc975611-2bb1-481c-b786-1399a8f95041-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.031040 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.031023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc975611-2bb1-481c-b786-1399a8f95041-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.036457 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.036431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc975611-2bb1-481c-b786-1399a8f95041-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.036549 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.036428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-995bb\" (UniqueName: \"kubernetes.io/projected/dc975611-2bb1-481c-b786-1399a8f95041-kube-api-access-995bb\") pod \"maas-default-gateway-openshift-default-56654dcf7-c4hrq\" (UID: \"dc975611-2bb1-481c-b786-1399a8f95041\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.178422 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.178328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:05.316533 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.316511    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"]
Jul 11 03:50:05.318611 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:50:05.318582    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc975611_2bb1_481c_b786_1399a8f95041.slice/crio-6cc6f1084b435294ca620395f2783793847264e34fef1531d5adb8a17ddb2af9 WatchSource:0}: Error finding container 6cc6f1084b435294ca620395f2783793847264e34fef1531d5adb8a17ddb2af9: Status 404 returned error can't find the container with id 6cc6f1084b435294ca620395f2783793847264e34fef1531d5adb8a17ddb2af9
Jul 11 03:50:05.611578 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:05.611538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" event={"ID":"dc975611-2bb1-481c-b786-1399a8f95041","Type":"ContainerStarted","Data":"6cc6f1084b435294ca620395f2783793847264e34fef1531d5adb8a17ddb2af9"}
Jul 11 03:50:06.937082 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:06.937049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:06.985118 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:06.985067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:08.026554 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:08.026521    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:50:08.026841 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:08.026587    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:50:08.026841 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:08.026615    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:50:08.628477 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:08.628445    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" event={"ID":"dc975611-2bb1-481c-b786-1399a8f95041","Type":"ContainerStarted","Data":"b182cf42212547558452abf5488bda79178685165d41fc78d4d98ed7ae77d861"}
Jul 11 03:50:08.651378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:08.651314    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" podStartSLOduration=1.945746467 podStartE2EDuration="4.651296366s" totalImagesPullingTime="2.705549899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:50:08.647259441 +0000 UTC m=+453.516706007" watchObservedRunningTime="2026-07-11 03:50:08.651296366 +0000 UTC m=+453.520742933"
Jul 11 03:50:09.179267 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.179231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:09.184361 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.184332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:09.634159 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.634130    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:09.635061 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.635040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:50:09.682772 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.682744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:09.690233 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.690208    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:09.739804 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.739771    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:09.739958 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.739865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:09.742186 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.742164    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 11 03:50:09.788415 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.788381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:09.795266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.795235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:09.852245 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.852205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:09.871184 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.871161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jrn\" (UniqueName: \"kubernetes.io/projected/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-kube-api-access-64jrn\") pod \"limitador-limitador-64c94fd64f-r9494\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:09.871269 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.871227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-config-file\") pod \"limitador-limitador-64c94fd64f-r9494\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:09.972338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.972246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-config-file\") pod \"limitador-limitador-64c94fd64f-r9494\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:09.972338 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.972326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jrn\" (UniqueName: \"kubernetes.io/projected/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-kube-api-access-64jrn\") pod \"limitador-limitador-64c94fd64f-r9494\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:09.972850 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.972830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-config-file\") pod \"limitador-limitador-64c94fd64f-r9494\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:09.980333 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:09.980308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jrn\" (UniqueName: \"kubernetes.io/projected/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-kube-api-access-64jrn\") pod \"limitador-limitador-64c94fd64f-r9494\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:10.050525 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.050489    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:10.194211 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.194188    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:10.196567 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:50:10.196542    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b99daa_31e7_4f8b_8d5e_6980c2b24e58.slice/crio-2b8ffa99111e17a29ccaa3fdff5e9589e1fb24a0af81741d076365cbf660424a WatchSource:0}: Error finding container 2b8ffa99111e17a29ccaa3fdff5e9589e1fb24a0af81741d076365cbf660424a: Status 404 returned error can't find the container with id 2b8ffa99111e17a29ccaa3fdff5e9589e1fb24a0af81741d076365cbf660424a
Jul 11 03:50:10.230138 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.229328    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:50:10.234156 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.234133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:50:10.234298 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.234231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.272489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.272462    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:50:10.377231 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.377195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3303dbfa-b6f9-4075-80fe-7e4546bbe087-config-file\") pod \"limitador-limitador-54d6d988d5-jsnps\" (UID: \"3303dbfa-b6f9-4075-80fe-7e4546bbe087\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.377390 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.377255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhv2\" (UniqueName: \"kubernetes.io/projected/3303dbfa-b6f9-4075-80fe-7e4546bbe087-kube-api-access-flhv2\") pod \"limitador-limitador-54d6d988d5-jsnps\" (UID: \"3303dbfa-b6f9-4075-80fe-7e4546bbe087\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.478170 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.478134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhv2\" (UniqueName: \"kubernetes.io/projected/3303dbfa-b6f9-4075-80fe-7e4546bbe087-kube-api-access-flhv2\") pod \"limitador-limitador-54d6d988d5-jsnps\" (UID: \"3303dbfa-b6f9-4075-80fe-7e4546bbe087\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.478331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.478219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3303dbfa-b6f9-4075-80fe-7e4546bbe087-config-file\") pod \"limitador-limitador-54d6d988d5-jsnps\" (UID: \"3303dbfa-b6f9-4075-80fe-7e4546bbe087\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.478835 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.478816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3303dbfa-b6f9-4075-80fe-7e4546bbe087-config-file\") pod \"limitador-limitador-54d6d988d5-jsnps\" (UID: \"3303dbfa-b6f9-4075-80fe-7e4546bbe087\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.501151 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.501089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhv2\" (UniqueName: \"kubernetes.io/projected/3303dbfa-b6f9-4075-80fe-7e4546bbe087-kube-api-access-flhv2\") pod \"limitador-limitador-54d6d988d5-jsnps\" (UID: \"3303dbfa-b6f9-4075-80fe-7e4546bbe087\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.545555 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.545525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:10.617301 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.617269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:10.641011 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.640971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494" event={"ID":"62b99daa-31e7-4f8b-8d5e-6980c2b24e58","Type":"ContainerStarted","Data":"2b8ffa99111e17a29ccaa3fdff5e9589e1fb24a0af81741d076365cbf660424a"}
Jul 11 03:50:10.681901 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.681877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:50:10.683862 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:50:10.683842    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3303dbfa_b6f9_4075_80fe_7e4546bbe087.slice/crio-9824681498aef2fa60e577d1981b6803b3b2f9fda9f5cc4a3885cecde0b76a4f WatchSource:0}: Error finding container 9824681498aef2fa60e577d1981b6803b3b2f9fda9f5cc4a3885cecde0b76a4f: Status 404 returned error can't find the container with id 9824681498aef2fa60e577d1981b6803b3b2f9fda9f5cc4a3885cecde0b76a4f
Jul 11 03:50:10.686654 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:10.686628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:11.648973 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:11.648928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps" event={"ID":"3303dbfa-b6f9-4075-80fe-7e4546bbe087","Type":"ContainerStarted","Data":"9824681498aef2fa60e577d1981b6803b3b2f9fda9f5cc4a3885cecde0b76a4f"}
Jul 11 03:50:12.137022 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:12.136990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:12.238150 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:12.238122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:13.660209 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:13.660170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494" event={"ID":"62b99daa-31e7-4f8b-8d5e-6980c2b24e58","Type":"ContainerStarted","Data":"8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88"}
Jul 11 03:50:13.660658 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:13.660279    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:13.661733 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:13.661713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps" event={"ID":"3303dbfa-b6f9-4075-80fe-7e4546bbe087","Type":"ContainerStarted","Data":"47ac930d2c48b7614af9f8c4de8ea174d17fe6936a7ed717ef63f28bb2e8fc49"}
Jul 11 03:50:13.661841 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:13.661759    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:13.675998 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:13.675952    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494" podStartSLOduration=1.7187575640000001 podStartE2EDuration="4.675938187s" totalImagesPullingTime="2.957180623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:50:13.67447847 +0000 UTC m=+458.543925036" watchObservedRunningTime="2026-07-11 03:50:13.675938187 +0000 UTC m=+458.545384878"
Jul 11 03:50:13.688914 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:13.688870    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps" podStartSLOduration=1.207812417 podStartE2EDuration="3.688858294s" totalImagesPullingTime="2.481045877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:50:13.688050081 +0000 UTC m=+458.557496644" watchObservedRunningTime="2026-07-11 03:50:13.688858294 +0000 UTC m=+458.558304857"
Jul 11 03:50:24.668063 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:24.668032    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsnps"
Jul 11 03:50:24.668576 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:24.668127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:24.724165 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:24.724135    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:24.724342 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:24.724300    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494" podUID="62b99daa-31e7-4f8b-8d5e-6980c2b24e58" containerName="limitador" containerID="cri-o://8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88" gracePeriod=30
Jul 11 03:50:25.283337 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.283279    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:25.408436 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.408409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-config-file\") pod \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") "
Jul 11 03:50:25.408592 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.408485    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64jrn\" (UniqueName: \"kubernetes.io/projected/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-kube-api-access-64jrn\") pod \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\" (UID: \"62b99daa-31e7-4f8b-8d5e-6980c2b24e58\") "
Jul 11 03:50:25.408830 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.408808    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-config-file" (OuterVolumeSpecName: "config-file") pod "62b99daa-31e7-4f8b-8d5e-6980c2b24e58" (UID: "62b99daa-31e7-4f8b-8d5e-6980c2b24e58"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 03:50:25.410659 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.410622    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-kube-api-access-64jrn" (OuterVolumeSpecName: "kube-api-access-64jrn") pod "62b99daa-31e7-4f8b-8d5e-6980c2b24e58" (UID: "62b99daa-31e7-4f8b-8d5e-6980c2b24e58"). InnerVolumeSpecName "kube-api-access-64jrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:50:25.509331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.509289    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jrn\" (UniqueName: \"kubernetes.io/projected/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-kube-api-access-64jrn\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:50:25.509331 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.509325    2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62b99daa-31e7-4f8b-8d5e-6980c2b24e58-config-file\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:50:25.721115 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.721058    2582 generic.go:356] "Generic (PLEG): container finished" podID="62b99daa-31e7-4f8b-8d5e-6980c2b24e58" containerID="8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88" exitCode=0
Jul 11 03:50:25.721584 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.721141    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494"
Jul 11 03:50:25.721584 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.721152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494" event={"ID":"62b99daa-31e7-4f8b-8d5e-6980c2b24e58","Type":"ContainerDied","Data":"8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88"}
Jul 11 03:50:25.721584 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.721256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-r9494" event={"ID":"62b99daa-31e7-4f8b-8d5e-6980c2b24e58","Type":"ContainerDied","Data":"2b8ffa99111e17a29ccaa3fdff5e9589e1fb24a0af81741d076365cbf660424a"}
Jul 11 03:50:25.721584 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.721290    2582 scope.go:117] "RemoveContainer" containerID="8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88"
Jul 11 03:50:25.730135 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.729952    2582 scope.go:117] "RemoveContainer" containerID="8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88"
Jul 11 03:50:25.730255 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:50:25.730235    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88\": container with ID starting with 8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88 not found: ID does not exist" containerID="8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88"
Jul 11 03:50:25.730302 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.730264    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88"} err="failed to get container status \"8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88\": rpc error: code = NotFound desc = could not find container \"8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88\": container with ID starting with 8c159dd410560198529275f07ade732fab883582b17afe325066a5643fccce88 not found: ID does not exist"
Jul 11 03:50:25.746888 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.746858    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:25.758496 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.758475    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-r9494"]
Jul 11 03:50:25.766513 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.766490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:25.867500 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:25.867454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:26.293160 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:26.293124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:27.481310 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:27.481279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:27.542264 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:27.542232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:27.645496 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:27.645464    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62b99daa-31e7-4f8b-8d5e-6980c2b24e58" path="/var/lib/kubelet/pods/62b99daa-31e7-4f8b-8d5e-6980c2b24e58/volumes"
Jul 11 03:50:37.853732 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:37.853697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:38.127541 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:38.127450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:44.106012 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.105976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:44.109074 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.109047    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fd464c864-q88z2"]
Jul 11 03:50:44.109425 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.109412    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b99daa-31e7-4f8b-8d5e-6980c2b24e58" containerName="limitador"
Jul 11 03:50:44.109470 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.109428    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b99daa-31e7-4f8b-8d5e-6980c2b24e58" containerName="limitador"
Jul 11 03:50:44.109507 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.109497    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b99daa-31e7-4f8b-8d5e-6980c2b24e58" containerName="limitador"
Jul 11 03:50:44.113828 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.113811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.116547 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.116527    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 11 03:50:44.116659 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.116589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9z8kq\""
Jul 11 03:50:44.116659 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.116611    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 11 03:50:44.122402 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.122379    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fd464c864-q88z2"]
Jul 11 03:50:44.165589 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.165560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.165749 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.165644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxqh\" (UniqueName: \"kubernetes.io/projected/4012af06-bf0c-44a3-90f3-59610720c729-kube-api-access-jkxqh\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.205966 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.205931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:44.266323 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.266288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxqh\" (UniqueName: \"kubernetes.io/projected/4012af06-bf0c-44a3-90f3-59610720c729-kube-api-access-jkxqh\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.266494 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.266339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.266494 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:50:44.266458    2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 11 03:50:44.266571 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:50:44.266518    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert podName:4012af06-bf0c-44a3-90f3-59610720c729 nodeName:}" failed. No retries permitted until 2026-07-11 03:50:44.766502273 +0000 UTC m=+489.635948815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert") pod "maas-controller-fd464c864-q88z2" (UID: "4012af06-bf0c-44a3-90f3-59610720c729") : secret "maas-controller-webhook-cert" not found
Jul 11 03:50:44.274297 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.274266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxqh\" (UniqueName: \"kubernetes.io/projected/4012af06-bf0c-44a3-90f3-59610720c729-kube-api-access-jkxqh\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.278481 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.278460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:44.282033 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.282014    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57c8d895cd-xsplt"]
Jul 11 03:50:44.284681 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.284664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.293317 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.293293    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-xsplt"]
Jul 11 03:50:44.367480 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.367390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6sq\" (UniqueName: \"kubernetes.io/projected/9661f4f9-5fdf-49ed-bf10-384fdf130316-kube-api-access-tx6sq\") pod \"maas-controller-57c8d895cd-xsplt\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") " pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.367480 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.367433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9661f4f9-5fdf-49ed-bf10-384fdf130316-webhook-cert\") pod \"maas-controller-57c8d895cd-xsplt\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") " pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.468452 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.468423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6sq\" (UniqueName: \"kubernetes.io/projected/9661f4f9-5fdf-49ed-bf10-384fdf130316-kube-api-access-tx6sq\") pod \"maas-controller-57c8d895cd-xsplt\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") " pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.468624 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.468460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9661f4f9-5fdf-49ed-bf10-384fdf130316-webhook-cert\") pod \"maas-controller-57c8d895cd-xsplt\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") " pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.470956 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.470935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9661f4f9-5fdf-49ed-bf10-384fdf130316-webhook-cert\") pod \"maas-controller-57c8d895cd-xsplt\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") " pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.476058 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.476022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:44.487431 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.487400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6sq\" (UniqueName: \"kubernetes.io/projected/9661f4f9-5fdf-49ed-bf10-384fdf130316-kube-api-access-tx6sq\") pod \"maas-controller-57c8d895cd-xsplt\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") " pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.595554 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.595525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:44.724519 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.724469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-xsplt"]
Jul 11 03:50:44.726685 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:50:44.726657    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9661f4f9_5fdf_49ed_bf10_384fdf130316.slice/crio-652407179b4840f460a2a2a9c4b495ca3a35ddb2f54da25656442077780c409e WatchSource:0}: Error finding container 652407179b4840f460a2a2a9c4b495ca3a35ddb2f54da25656442077780c409e: Status 404 returned error can't find the container with id 652407179b4840f460a2a2a9c4b495ca3a35ddb2f54da25656442077780c409e
Jul 11 03:50:44.771156 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.771122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.773558 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.773531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert\") pod \"maas-controller-fd464c864-q88z2\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") " pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:44.809716 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:44.809680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c8d895cd-xsplt" event={"ID":"9661f4f9-5fdf-49ed-bf10-384fdf130316","Type":"ContainerStarted","Data":"652407179b4840f460a2a2a9c4b495ca3a35ddb2f54da25656442077780c409e"}
Jul 11 03:50:45.025986 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:45.025946    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:45.157162 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:45.157138    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fd464c864-q88z2"]
Jul 11 03:50:45.159353 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:50:45.159330    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4012af06_bf0c_44a3_90f3_59610720c729.slice/crio-f562eb5d2ea38cf555d454ee78e3488a41f2d3d13f0f038e6ff1a90617953491 WatchSource:0}: Error finding container f562eb5d2ea38cf555d454ee78e3488a41f2d3d13f0f038e6ff1a90617953491: Status 404 returned error can't find the container with id f562eb5d2ea38cf555d454ee78e3488a41f2d3d13f0f038e6ff1a90617953491
Jul 11 03:50:45.819067 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:45.819023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fd464c864-q88z2" event={"ID":"4012af06-bf0c-44a3-90f3-59610720c729","Type":"ContainerStarted","Data":"f562eb5d2ea38cf555d454ee78e3488a41f2d3d13f0f038e6ff1a90617953491"}
Jul 11 03:50:47.127203 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.127173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:47.227713 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.227615    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:50:47.835225 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.835192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fd464c864-q88z2" event={"ID":"4012af06-bf0c-44a3-90f3-59610720c729","Type":"ContainerStarted","Data":"4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef"}
Jul 11 03:50:47.835410 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.835294    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:47.836774 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.836750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c8d895cd-xsplt" event={"ID":"9661f4f9-5fdf-49ed-bf10-384fdf130316","Type":"ContainerStarted","Data":"627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6"}
Jul 11 03:50:47.836893 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.836822    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:47.852371 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.852325    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fd464c864-q88z2" podStartSLOduration=2.080798919 podStartE2EDuration="3.852310182s" totalImagesPullingTime="1.771511263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:50:47.850692301 +0000 UTC m=+492.720138901" watchObservedRunningTime="2026-07-11 03:50:47.852310182 +0000 UTC m=+492.721756745"
Jul 11 03:50:47.868926 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:47.868872    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57c8d895cd-xsplt" podStartSLOduration=1.674740167 podStartE2EDuration="3.868856457s" totalImagesPullingTime="2.19411629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:50:47.867203828 +0000 UTC m=+492.736650391" watchObservedRunningTime="2026-07-11 03:50:47.868856457 +0000 UTC m=+492.738303024"
Jul 11 03:50:58.842829 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:58.842794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:58.843336 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:58.843049    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:50:58.905894 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:58.905860    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fd464c864-q88z2"]
Jul 11 03:50:58.906064 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:58.906042    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fd464c864-q88z2" podUID="4012af06-bf0c-44a3-90f3-59610720c729" containerName="manager" containerID="cri-o://4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef" gracePeriod=30
Jul 11 03:50:59.154521 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.154500    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:59.201708 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.201681    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkxqh\" (UniqueName: \"kubernetes.io/projected/4012af06-bf0c-44a3-90f3-59610720c729-kube-api-access-jkxqh\") pod \"4012af06-bf0c-44a3-90f3-59610720c729\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") "
Jul 11 03:50:59.201889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.201742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert\") pod \"4012af06-bf0c-44a3-90f3-59610720c729\" (UID: \"4012af06-bf0c-44a3-90f3-59610720c729\") "
Jul 11 03:50:59.204058 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.204024    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4012af06-bf0c-44a3-90f3-59610720c729" (UID: "4012af06-bf0c-44a3-90f3-59610720c729"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:50:59.204187 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.204074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4012af06-bf0c-44a3-90f3-59610720c729-kube-api-access-jkxqh" (OuterVolumeSpecName: "kube-api-access-jkxqh") pod "4012af06-bf0c-44a3-90f3-59610720c729" (UID: "4012af06-bf0c-44a3-90f3-59610720c729"). InnerVolumeSpecName "kube-api-access-jkxqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:50:59.303069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.303025    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4012af06-bf0c-44a3-90f3-59610720c729-webhook-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:50:59.303069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.303058    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxqh\" (UniqueName: \"kubernetes.io/projected/4012af06-bf0c-44a3-90f3-59610720c729-kube-api-access-jkxqh\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:50:59.891086 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.891053    2582 generic.go:356] "Generic (PLEG): container finished" podID="4012af06-bf0c-44a3-90f3-59610720c729" containerID="4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef" exitCode=0
Jul 11 03:50:59.891086 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.891110    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fd464c864-q88z2"
Jul 11 03:50:59.891603 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.891116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fd464c864-q88z2" event={"ID":"4012af06-bf0c-44a3-90f3-59610720c729","Type":"ContainerDied","Data":"4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef"}
Jul 11 03:50:59.891603 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.891158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fd464c864-q88z2" event={"ID":"4012af06-bf0c-44a3-90f3-59610720c729","Type":"ContainerDied","Data":"f562eb5d2ea38cf555d454ee78e3488a41f2d3d13f0f038e6ff1a90617953491"}
Jul 11 03:50:59.891603 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.891181    2582 scope.go:117] "RemoveContainer" containerID="4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef"
Jul 11 03:50:59.899443 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.899413    2582 scope.go:117] "RemoveContainer" containerID="4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef"
Jul 11 03:50:59.899696 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:50:59.899675    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef\": container with ID starting with 4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef not found: ID does not exist" containerID="4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef"
Jul 11 03:50:59.899754 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.899705    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef"} err="failed to get container status \"4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef\": rpc error: code = NotFound desc = could not find container \"4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef\": container with ID starting with 4a7ee82600696de0a410f11fcf35d9df7128cf0a9b63d3b442a8e630c801ecef not found: ID does not exist"
Jul 11 03:50:59.906291 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.906237    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fd464c864-q88z2"]
Jul 11 03:50:59.908533 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:50:59.908514    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fd464c864-q88z2"]
Jul 11 03:51:01.646213 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:01.646183    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4012af06-bf0c-44a3-90f3-59610720c729" path="/var/lib/kubelet/pods/4012af06-bf0c-44a3-90f3-59610720c729/volumes"
Jul 11 03:51:19.060091 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.060054    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:19.155028 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.154995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:19.496041 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.496001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:19.626152 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.626116    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"]
Jul 11 03:51:19.627135 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.627091    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4012af06-bf0c-44a3-90f3-59610720c729" containerName="manager"
Jul 11 03:51:19.627277 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.627266    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4012af06-bf0c-44a3-90f3-59610720c729" containerName="manager"
Jul 11 03:51:19.627554 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.627540    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4012af06-bf0c-44a3-90f3-59610720c729" containerName="manager"
Jul 11 03:51:19.634558 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.634353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.636858 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.636832    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"]
Jul 11 03:51:19.636989 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.636972    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 11 03:51:19.637063 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.637002    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 11 03:51:19.637611 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.637597    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h78n4\""
Jul 11 03:51:19.637611 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.637606    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 11 03:51:19.671031 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.670998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:19.789935 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.789858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc66443d-cca6-4b5b-973b-0b31187a5ef7-maas-api-tls\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.789935 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.789889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc66443d-cca6-4b5b-973b-0b31187a5ef7-openshift-service-ca\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.789935 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.789913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g589\" (UniqueName: \"kubernetes.io/projected/cc66443d-cca6-4b5b-973b-0b31187a5ef7-kube-api-access-9g589\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.891522 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.891477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc66443d-cca6-4b5b-973b-0b31187a5ef7-maas-api-tls\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.891522 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.891527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc66443d-cca6-4b5b-973b-0b31187a5ef7-openshift-service-ca\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.891791 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.891566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g589\" (UniqueName: \"kubernetes.io/projected/cc66443d-cca6-4b5b-973b-0b31187a5ef7-kube-api-access-9g589\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.892278 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.892256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc66443d-cca6-4b5b-973b-0b31187a5ef7-openshift-service-ca\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.894339 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.894308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc66443d-cca6-4b5b-973b-0b31187a5ef7-maas-api-tls\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.899684 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.899660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g589\" (UniqueName: \"kubernetes.io/projected/cc66443d-cca6-4b5b-973b-0b31187a5ef7-kube-api-access-9g589\") pod \"maas-api-6c6b47654b-8vfr6\" (UID: \"cc66443d-cca6-4b5b-973b-0b31187a5ef7\") " pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:19.946492 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:19.946464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:20.064620 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.064497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:20.099612 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.099494    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"]
Jul 11 03:51:20.101964 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:51:20.101936    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc66443d_cca6_4b5b_973b_0b31187a5ef7.slice/crio-f4a012385a3ac2c037eb0b88766d9087352f0d62dd7803d0d0c60655aa95c2a4 WatchSource:0}: Error finding container f4a012385a3ac2c037eb0b88766d9087352f0d62dd7803d0d0c60655aa95c2a4: Status 404 returned error can't find the container with id f4a012385a3ac2c037eb0b88766d9087352f0d62dd7803d0d0c60655aa95c2a4
Jul 11 03:51:20.121387 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.121355    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:51:20.130974 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.130950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:20.191544 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.191509    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-crhmt"]
Jul 11 03:51:20.197197 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.197174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.199458 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.199429    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 11 03:51:20.199458 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.199431    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4qpkd\""
Jul 11 03:51:20.205167 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.205144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-crhmt"]
Jul 11 03:51:20.295682 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.295648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrmh\" (UniqueName: \"kubernetes.io/projected/10a64279-544a-4e5f-a57c-43f3bf17b598-kube-api-access-7zrmh\") pod \"payload-processing-f9b8d9b9b-crhmt\" (UID: \"10a64279-544a-4e5f-a57c-43f3bf17b598\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.295871 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.295746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a64279-544a-4e5f-a57c-43f3bf17b598-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-crhmt\" (UID: \"10a64279-544a-4e5f-a57c-43f3bf17b598\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.396761 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.396675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrmh\" (UniqueName: \"kubernetes.io/projected/10a64279-544a-4e5f-a57c-43f3bf17b598-kube-api-access-7zrmh\") pod \"payload-processing-f9b8d9b9b-crhmt\" (UID: \"10a64279-544a-4e5f-a57c-43f3bf17b598\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.396761 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.396725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a64279-544a-4e5f-a57c-43f3bf17b598-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-crhmt\" (UID: \"10a64279-544a-4e5f-a57c-43f3bf17b598\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.397492 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.397471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a64279-544a-4e5f-a57c-43f3bf17b598-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-crhmt\" (UID: \"10a64279-544a-4e5f-a57c-43f3bf17b598\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.404727 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.404707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrmh\" (UniqueName: \"kubernetes.io/projected/10a64279-544a-4e5f-a57c-43f3bf17b598-kube-api-access-7zrmh\") pod \"payload-processing-f9b8d9b9b-crhmt\" (UID: \"10a64279-544a-4e5f-a57c-43f3bf17b598\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.509729 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.509694    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:20.649333 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.649309    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-crhmt"]
Jul 11 03:51:20.651695 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:51:20.651659    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a64279_544a_4e5f_a57c_43f3bf17b598.slice/crio-078d1ce0cb954abb318a90207786b9173f99bcd23ffa3040ff3ec4a45dae0c32 WatchSource:0}: Error finding container 078d1ce0cb954abb318a90207786b9173f99bcd23ffa3040ff3ec4a45dae0c32: Status 404 returned error can't find the container with id 078d1ce0cb954abb318a90207786b9173f99bcd23ffa3040ff3ec4a45dae0c32
Jul 11 03:51:20.989396 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.989321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6" event={"ID":"cc66443d-cca6-4b5b-973b-0b31187a5ef7","Type":"ContainerStarted","Data":"f4a012385a3ac2c037eb0b88766d9087352f0d62dd7803d0d0c60655aa95c2a4"}
Jul 11 03:51:20.991826 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:20.991790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt" event={"ID":"10a64279-544a-4e5f-a57c-43f3bf17b598","Type":"ContainerStarted","Data":"078d1ce0cb954abb318a90207786b9173f99bcd23ffa3040ff3ec4a45dae0c32"}
Jul 11 03:51:23.003785 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:23.003751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6" event={"ID":"cc66443d-cca6-4b5b-973b-0b31187a5ef7","Type":"ContainerStarted","Data":"518b94d4bf704510a32c5257c8294655c7362ab8bdf091356daa249d30937463"}
Jul 11 03:51:23.004227 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:23.003863    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:23.005240 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:23.005217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt" event={"ID":"10a64279-544a-4e5f-a57c-43f3bf17b598","Type":"ContainerStarted","Data":"1240986f258750ee43f1824389ed59b16a51a59a37bbe5c0aae34d5832ab8564"}
Jul 11 03:51:23.005344 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:23.005268    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:23.020625 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:23.020587    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6" podStartSLOduration=1.53184077 podStartE2EDuration="4.020572359s" totalImagesPullingTime="2.488731589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:51:23.019183609 +0000 UTC m=+527.888630173" watchObservedRunningTime="2026-07-11 03:51:23.020572359 +0000 UTC m=+527.890018922"
Jul 11 03:51:23.035240 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:23.035199    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt" podStartSLOduration=1.094709946 podStartE2EDuration="3.035183513s" totalImagesPullingTime="1.940473567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:51:23.034447785 +0000 UTC m=+527.903894380" watchObservedRunningTime="2026-07-11 03:51:23.035183513 +0000 UTC m=+527.904630077"
Jul 11 03:51:28.031165 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:28.031133    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc975611-2bb1-481c-b786-1399a8f95041" containerID="b182cf42212547558452abf5488bda79178685165d41fc78d4d98ed7ae77d861" exitCode=0
Jul 11 03:51:28.031700 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:28.031218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" event={"ID":"dc975611-2bb1-481c-b786-1399a8f95041","Type":"ContainerDied","Data":"b182cf42212547558452abf5488bda79178685165d41fc78d4d98ed7ae77d861"}
Jul 11 03:51:28.031767 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:28.031731    2582 scope.go:117] "RemoveContainer" containerID="b182cf42212547558452abf5488bda79178685165d41fc78d4d98ed7ae77d861"
Jul 11 03:51:28.032714 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:28.032687    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:51:28.032776 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:28.032753    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:51:28.032818 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:28.032786    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 11 03:51:29.016089 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:29.016058    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c6b47654b-8vfr6"
Jul 11 03:51:29.044075 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:29.044037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" event={"ID":"dc975611-2bb1-481c-b786-1399a8f95041","Type":"ContainerStarted","Data":"1cfb897374675e00492f228d640b6236d7cb0e8e6b3bca9986b7e81fee0eb4d5"}
Jul 11 03:51:29.179088 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:29.179045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:51:29.179318 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:29.179292    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-c4hrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 11 03:51:29.179415 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:29.179370    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" podUID="dc975611-2bb1-481c-b786-1399a8f95041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 11 03:51:30.178797 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:30.178759    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-c4hrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 11 03:51:30.179272 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:30.178837    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" podUID="dc975611-2bb1-481c-b786-1399a8f95041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 11 03:51:30.429670 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:30.429556    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:30.540921 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:30.540878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:31.179225 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:31.179186    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-c4hrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 11 03:51:31.179639 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:31.179254    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" podUID="dc975611-2bb1-481c-b786-1399a8f95041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 11 03:51:32.179137 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:32.179069    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-c4hrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 11 03:51:32.179321 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:32.179151    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq" podUID="dc975611-2bb1-481c-b786-1399a8f95041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 11 03:51:33.182032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:33.182005    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:51:33.182489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:33.182281    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:51:33.182950 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:33.182933    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-c4hrq"
Jul 11 03:51:34.012848 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:34.012809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-crhmt"
Jul 11 03:51:46.163728 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:46.163689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:53.496793 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:53.496761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:57.725239 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.725203    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"]
Jul 11 03:51:57.728433 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.728415    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.730925 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.730900    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 11 03:51:57.730925 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.730921    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 11 03:51:57.731767 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.731747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lvfkx\""
Jul 11 03:51:57.731938 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.731868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 11 03:51:57.737113 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.737081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"]
Jul 11 03:51:57.756340 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.756294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:51:57.823443 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.823403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.823443 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.823443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.823675 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.823475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28n6\" (UniqueName: \"kubernetes.io/projected/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-kube-api-access-f28n6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.823675 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.823525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.823675 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.823636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.823793 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.823714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924199 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28n6\" (UniqueName: \"kubernetes.io/projected/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-kube-api-access-f28n6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924378 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924674 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924734 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.924878 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.924856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.926866 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.926844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.927087 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.927066    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:57.932256 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:57.932234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28n6\" (UniqueName: \"kubernetes.io/projected/d675d7d0-9d4c-4121-a0df-824ebb60ff7e-kube-api-access-f28n6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l\" (UID: \"d675d7d0-9d4c-4121-a0df-824ebb60ff7e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:58.039703 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:58.039664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:51:58.195437 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:58.195415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"]
Jul 11 03:51:58.197249 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:51:58.197224    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd675d7d0_9d4c_4121_a0df_824ebb60ff7e.slice/crio-ae913ee9664dab94e48631d04267a159f6ff48ec330aa1d8332a33af58e47f34 WatchSource:0}: Error finding container ae913ee9664dab94e48631d04267a159f6ff48ec330aa1d8332a33af58e47f34: Status 404 returned error can't find the container with id ae913ee9664dab94e48631d04267a159f6ff48ec330aa1d8332a33af58e47f34
Jul 11 03:51:58.302275 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:58.302185    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:51:59.195838 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:51:59.195803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l" event={"ID":"d675d7d0-9d4c-4121-a0df-824ebb60ff7e","Type":"ContainerStarted","Data":"ae913ee9664dab94e48631d04267a159f6ff48ec330aa1d8332a33af58e47f34"}
Jul 11 03:52:00.202793 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:00.202756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l" event={"ID":"d675d7d0-9d4c-4121-a0df-824ebb60ff7e","Type":"ContainerStarted","Data":"a24f33d6b26afe7ab3894519d3a30a95a9651b3ee21ee5dba0302597c49ccab7"}
Jul 11 03:52:00.219512 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:00.219461    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l" podStartSLOduration=2.118290514 podStartE2EDuration="3.21943414s" totalImagesPullingTime="1.101143626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:52:00.218034369 +0000 UTC m=+565.087480934" watchObservedRunningTime="2026-07-11 03:52:00.21943414 +0000 UTC m=+565.088880682"
Jul 11 03:52:08.040040 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:08.039992    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:52:08.040540 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:08.040062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:52:08.052790 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:08.052763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:52:08.251582 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:08.251554    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l"
Jul 11 03:52:13.075185 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:13.075144    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:13.799490 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:13.799451    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:52:20.341412 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:20.341376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:21.496072 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:21.496041    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:52:23.601650 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.601615    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"]
Jul 11 03:52:23.606140 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.606091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.608480 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.608458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 11 03:52:23.614390 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.614364    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"]
Jul 11 03:52:23.625185 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.625086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:23.752455 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.752408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.752643 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.752466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0400edf2-cc87-4ee3-a035-1c91b08c22e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.752643 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.752536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.752740 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.752630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.752740 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.752669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.752827 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.752784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvg9s\" (UniqueName: \"kubernetes.io/projected/0400edf2-cc87-4ee3-a035-1c91b08c22e0-kube-api-access-vvg9s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.853861 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.853766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvg9s\" (UniqueName: \"kubernetes.io/projected/0400edf2-cc87-4ee3-a035-1c91b08c22e0-kube-api-access-vvg9s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.853861 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.853836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854115 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.853866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0400edf2-cc87-4ee3-a035-1c91b08c22e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854115 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.853909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854302 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.854135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854302 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.854191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854625 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.854584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854625 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.854591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.854853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.854834    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.857139 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.857089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0400edf2-cc87-4ee3-a035-1c91b08c22e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.857299 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.857281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0400edf2-cc87-4ee3-a035-1c91b08c22e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.861317 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.861293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvg9s\" (UniqueName: \"kubernetes.io/projected/0400edf2-cc87-4ee3-a035-1c91b08c22e0-kube-api-access-vvg9s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm\" (UID: \"0400edf2-cc87-4ee3-a035-1c91b08c22e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:23.918847 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:23.918819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:24.061853 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:24.061823    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"]
Jul 11 03:52:24.066756 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:52:24.066718    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0400edf2_cc87_4ee3_a035_1c91b08c22e0.slice/crio-2ee87af810bf5ae9f12420837b71337fe560214deba3b38c5b8fd23b2196d2b7 WatchSource:0}: Error finding container 2ee87af810bf5ae9f12420837b71337fe560214deba3b38c5b8fd23b2196d2b7: Status 404 returned error can't find the container with id 2ee87af810bf5ae9f12420837b71337fe560214deba3b38c5b8fd23b2196d2b7
Jul 11 03:52:24.318744 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:24.318711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm" event={"ID":"0400edf2-cc87-4ee3-a035-1c91b08c22e0","Type":"ContainerStarted","Data":"7e38a4fc3619f6b9b44a98b551a227fb617c551ab4dfe8ddc316373449e38fcf"}
Jul 11 03:52:24.318949 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:24.318767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm" event={"ID":"0400edf2-cc87-4ee3-a035-1c91b08c22e0","Type":"ContainerStarted","Data":"2ee87af810bf5ae9f12420837b71337fe560214deba3b38c5b8fd23b2196d2b7"}
Jul 11 03:52:24.337024 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:24.336971    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm" podStartSLOduration=1.180196507 podStartE2EDuration="1.336948691s" totalImagesPullingTime="156.752184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:52:24.33433263 +0000 UTC m=+589.203779194" watchObservedRunningTime="2026-07-11 03:52:24.336948691 +0000 UTC m=+589.206395258"
Jul 11 03:52:25.412872 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:25.412836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:52:33.919684 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:33.919646    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:33.920140 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:33.919723    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:33.932857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:33.932829    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:34.376384 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:34.376357    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm"
Jul 11 03:52:35.718557 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:35.718527    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:52:35.719765 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:35.719743    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:52:37.235064 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:37.235030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:39.103851 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:39.103819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:52:47.242926 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:47.242887    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:48.328435 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:48.328399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:52.103357 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.103327    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"]
Jul 11 03:52:52.108749 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.108725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.111224 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.111191    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 11 03:52:52.118044 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.118014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"]
Jul 11 03:52:52.141873 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.141836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:52:52.205916 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.205890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.206069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.205923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.206069 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.206022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.206166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.206070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.206166 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.206151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.206232 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.206188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vr22\" (UniqueName: \"kubernetes.io/projected/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-kube-api-access-6vr22\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.306943 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.306910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.306943 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.306952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.306992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.307026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vr22\" (UniqueName: \"kubernetes.io/projected/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-kube-api-access-6vr22\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.307066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307266 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.307178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307464 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.307382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307464 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.307400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.307565 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.307461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.309539 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.309512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.309894 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.309874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.314411 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.314392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vr22\" (UniqueName: \"kubernetes.io/projected/1e7024fd-fc12-4baa-8e93-5eb9b0d9f114-kube-api-access-6vr22\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz\" (UID: \"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.420037 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.419943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:52:52.557440 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.557413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"]
Jul 11 03:52:52.559581 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:52:52.559544    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e7024fd_fc12_4baa_8e93_5eb9b0d9f114.slice/crio-d70f6c4a087d9571651fbff9c24a37ca0caf409a8e703538437f7054002b49c6 WatchSource:0}: Error finding container d70f6c4a087d9571651fbff9c24a37ca0caf409a8e703538437f7054002b49c6: Status 404 returned error can't find the container with id d70f6c4a087d9571651fbff9c24a37ca0caf409a8e703538437f7054002b49c6
Jul 11 03:52:52.561345 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:52.561324    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 03:52:53.457186 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:53.457145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz" event={"ID":"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114","Type":"ContainerStarted","Data":"664f140d44cdec2032789ad3d2283456c5d992667933f28d97e5ed826cb42d83"}
Jul 11 03:52:53.457186 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:53.457191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz" event={"ID":"1e7024fd-fc12-4baa-8e93-5eb9b0d9f114","Type":"ContainerStarted","Data":"d70f6c4a087d9571651fbff9c24a37ca0caf409a8e703538437f7054002b49c6"}
Jul 11 03:52:53.472746 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:53.472697    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz" podStartSLOduration=1.301628559 podStartE2EDuration="1.472682054s" totalImagesPullingTime="171.053495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:52:53.472249539 +0000 UTC m=+618.341696116" watchObservedRunningTime="2026-07-11 03:52:53.472682054 +0000 UTC m=+618.342128617"
Jul 11 03:52:54.611221 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:54.611182    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:52:58.327731 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:52:58.327654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:53:02.420919 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:02.420880    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:53:02.420919 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:02.420932    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:53:02.433424 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:02.433396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:53:02.515031 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:02.515003    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz"
Jul 11 03:53:55.064076 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:55.064042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:53:55.159350 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:55.159313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:53:56.176840 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:56.176800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:53:56.269971 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:53:56.269942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:54:23.612925 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:54:23.612889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:54:33.445494 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:54:33.445414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:54:44.005963 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:54:44.005926    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:54:53.313497 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:54:53.313458    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:54:58.256029 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:54:58.255987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:05.008369 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:05.008334    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:55:06.130878 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.130840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:06.235077 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.235043    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-xsplt"]
Jul 11 03:55:06.235270 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.235249    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57c8d895cd-xsplt" podUID="9661f4f9-5fdf-49ed-bf10-384fdf130316" containerName="manager" containerID="cri-o://627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6" gracePeriod=30
Jul 11 03:55:06.495523 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.495498    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:55:06.604534 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.604505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx6sq\" (UniqueName: \"kubernetes.io/projected/9661f4f9-5fdf-49ed-bf10-384fdf130316-kube-api-access-tx6sq\") pod \"9661f4f9-5fdf-49ed-bf10-384fdf130316\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") "
Jul 11 03:55:06.604692 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.604570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9661f4f9-5fdf-49ed-bf10-384fdf130316-webhook-cert\") pod \"9661f4f9-5fdf-49ed-bf10-384fdf130316\" (UID: \"9661f4f9-5fdf-49ed-bf10-384fdf130316\") "
Jul 11 03:55:06.606766 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.606734    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9661f4f9-5fdf-49ed-bf10-384fdf130316-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9661f4f9-5fdf-49ed-bf10-384fdf130316" (UID: "9661f4f9-5fdf-49ed-bf10-384fdf130316"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 03:55:06.606913 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.606791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9661f4f9-5fdf-49ed-bf10-384fdf130316-kube-api-access-tx6sq" (OuterVolumeSpecName: "kube-api-access-tx6sq") pod "9661f4f9-5fdf-49ed-bf10-384fdf130316" (UID: "9661f4f9-5fdf-49ed-bf10-384fdf130316"). InnerVolumeSpecName "kube-api-access-tx6sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:55:06.705875 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.705789    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9661f4f9-5fdf-49ed-bf10-384fdf130316-webhook-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:55:06.705875 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:06.705820    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx6sq\" (UniqueName: \"kubernetes.io/projected/9661f4f9-5fdf-49ed-bf10-384fdf130316-kube-api-access-tx6sq\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:55:07.100843 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.100808    2582 generic.go:356] "Generic (PLEG): container finished" podID="9661f4f9-5fdf-49ed-bf10-384fdf130316" containerID="627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6" exitCode=0
Jul 11 03:55:07.100999 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.100862    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c8d895cd-xsplt"
Jul 11 03:55:07.100999 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.100895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c8d895cd-xsplt" event={"ID":"9661f4f9-5fdf-49ed-bf10-384fdf130316","Type":"ContainerDied","Data":"627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6"}
Jul 11 03:55:07.100999 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.100952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c8d895cd-xsplt" event={"ID":"9661f4f9-5fdf-49ed-bf10-384fdf130316","Type":"ContainerDied","Data":"652407179b4840f460a2a2a9c4b495ca3a35ddb2f54da25656442077780c409e"}
Jul 11 03:55:07.100999 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.100980    2582 scope.go:117] "RemoveContainer" containerID="627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6"
Jul 11 03:55:07.110348 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.110326    2582 scope.go:117] "RemoveContainer" containerID="627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6"
Jul 11 03:55:07.110611 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:55:07.110593    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6\": container with ID starting with 627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6 not found: ID does not exist" containerID="627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6"
Jul 11 03:55:07.110657 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.110620    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6"} err="failed to get container status \"627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6\": rpc error: code = NotFound desc = could not find container \"627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6\": container with ID starting with 627d53bc269c6e0976dff866c0009a46a0c7099a14c57a15595f4d368c21acc6 not found: ID does not exist"
Jul 11 03:55:07.120851 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.120829    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-xsplt"]
Jul 11 03:55:07.123716 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.123695    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-xsplt"]
Jul 11 03:55:07.646252 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.646222    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9661f4f9-5fdf-49ed-bf10-384fdf130316" path="/var/lib/kubelet/pods/9661f4f9-5fdf-49ed-bf10-384fdf130316/volumes"
Jul 11 03:55:07.761882 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.761843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:07.884810 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.884773    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57c8d895cd-lmmpf"]
Jul 11 03:55:07.885293 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.885278    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9661f4f9-5fdf-49ed-bf10-384fdf130316" containerName="manager"
Jul 11 03:55:07.885348 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.885295    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9661f4f9-5fdf-49ed-bf10-384fdf130316" containerName="manager"
Jul 11 03:55:07.885385 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.885365    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9661f4f9-5fdf-49ed-bf10-384fdf130316" containerName="manager"
Jul 11 03:55:07.889724 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.889705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:07.891939 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.891922    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 11 03:55:07.892826 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.892805    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 11 03:55:07.892874 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.892809    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9z8kq\""
Jul 11 03:55:07.898680 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:07.898659    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-lmmpf"]
Jul 11 03:55:08.017640 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.017606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e03d8538-d22e-4b32-b4af-879a2d0b3704-webhook-cert\") pod \"maas-controller-57c8d895cd-lmmpf\" (UID: \"e03d8538-d22e-4b32-b4af-879a2d0b3704\") " pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.017826 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.017742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv59w\" (UniqueName: \"kubernetes.io/projected/e03d8538-d22e-4b32-b4af-879a2d0b3704-kube-api-access-bv59w\") pod \"maas-controller-57c8d895cd-lmmpf\" (UID: \"e03d8538-d22e-4b32-b4af-879a2d0b3704\") " pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.119005 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.118976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e03d8538-d22e-4b32-b4af-879a2d0b3704-webhook-cert\") pod \"maas-controller-57c8d895cd-lmmpf\" (UID: \"e03d8538-d22e-4b32-b4af-879a2d0b3704\") " pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.119247 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.119038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv59w\" (UniqueName: \"kubernetes.io/projected/e03d8538-d22e-4b32-b4af-879a2d0b3704-kube-api-access-bv59w\") pod \"maas-controller-57c8d895cd-lmmpf\" (UID: \"e03d8538-d22e-4b32-b4af-879a2d0b3704\") " pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.121684 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.121665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e03d8538-d22e-4b32-b4af-879a2d0b3704-webhook-cert\") pod \"maas-controller-57c8d895cd-lmmpf\" (UID: \"e03d8538-d22e-4b32-b4af-879a2d0b3704\") " pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.127406 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.127385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv59w\" (UniqueName: \"kubernetes.io/projected/e03d8538-d22e-4b32-b4af-879a2d0b3704-kube-api-access-bv59w\") pod \"maas-controller-57c8d895cd-lmmpf\" (UID: \"e03d8538-d22e-4b32-b4af-879a2d0b3704\") " pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.201146 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.201054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:08.346058 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:08.342665    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c8d895cd-lmmpf"]
Jul 11 03:55:08.347834 ip-10-0-140-230 kubenswrapper[2582]: W0711 03:55:08.347800    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03d8538_d22e_4b32_b4af_879a2d0b3704.slice/crio-e106bfc4375c6e5404dbe2555851407a4783c638b18b4abf24b485a9ea0a43e7 WatchSource:0}: Error finding container e106bfc4375c6e5404dbe2555851407a4783c638b18b4abf24b485a9ea0a43e7: Status 404 returned error can't find the container with id e106bfc4375c6e5404dbe2555851407a4783c638b18b4abf24b485a9ea0a43e7
Jul 11 03:55:09.113359 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:09.113264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c8d895cd-lmmpf" event={"ID":"e03d8538-d22e-4b32-b4af-879a2d0b3704","Type":"ContainerStarted","Data":"dfe76057cdc8d24ecb04f83f40c16270c9f58d74af5040e8a83ddc02f64749a4"}
Jul 11 03:55:09.113359 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:09.113328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c8d895cd-lmmpf" event={"ID":"e03d8538-d22e-4b32-b4af-879a2d0b3704","Type":"ContainerStarted","Data":"e106bfc4375c6e5404dbe2555851407a4783c638b18b4abf24b485a9ea0a43e7"}
Jul 11 03:55:09.113744 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:09.113392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:09.129827 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:09.129785    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57c8d895cd-lmmpf" podStartSLOduration=1.646623714 podStartE2EDuration="2.129771209s" totalImagesPullingTime="483.147495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:55:09.127866174 +0000 UTC m=+753.997312739" watchObservedRunningTime="2026-07-11 03:55:09.129771209 +0000 UTC m=+753.999217774"
Jul 11 03:55:20.120050 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:20.120016    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57c8d895cd-lmmpf"
Jul 11 03:55:30.114005 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:30.113970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:55:42.304783 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:42.304751    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:55:51.317087 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:51.317038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:51.419367 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:51.419330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:51.579121 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:51.579014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:57.307478 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:57.307388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:57.371626 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:57.371577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:57.446152 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:57.446081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:57.545585 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:57.545547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:55:57.649113 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:55:57.649016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:03.410328 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:03.410277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:03.466743 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:03.466708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:08.120755 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:08.120723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:08.222144 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:08.222110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:20.512681 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:20.512645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:56:30.004427 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:30.004378    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:56:36.560544 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:36.560507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:36.609889 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:36.609857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 03:56:40.604498 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:40.604467    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:56:48.412192 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:48.412157    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:56:58.308353 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:56:58.308318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:57:14.206768 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:57:14.206734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:57:21.005857 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:57:21.005822    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:57:28.809293 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:57:28.809197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:57:35.753382 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:57:35.753346    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:57:35.755123 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:57:35.755078    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 03:57:51.301022 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:57:51.300978    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:58:03.306259 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:58:03.306223    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:58:25.603122 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:58:25.603059    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:58:36.104513 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:58:36.104469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:58:51.804868 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:58:51.804834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:59:01.409148 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:01.409043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:59:10.806332 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:10.806297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:59:20.503032 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:20.502995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:59:29.114543 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:29.114504    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 03:59:55.662315 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:55.662274    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"]
Jul 11 03:59:55.662945 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:55.662614    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt" podUID="c62c474b-fc0f-4db3-a584-a8b6a705fb65" containerName="manager" containerID="cri-o://72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01" gracePeriod=10
Jul 11 03:59:59.326067 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.326042    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:59:59.518323 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.518292    2582 generic.go:356] "Generic (PLEG): container finished" podID="c62c474b-fc0f-4db3-a584-a8b6a705fb65" containerID="72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01" exitCode=0
Jul 11 03:59:59.518489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.518347    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"
Jul 11 03:59:59.518489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.518377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt" event={"ID":"c62c474b-fc0f-4db3-a584-a8b6a705fb65","Type":"ContainerDied","Data":"72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01"}
Jul 11 03:59:59.518489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.518448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt" event={"ID":"c62c474b-fc0f-4db3-a584-a8b6a705fb65","Type":"ContainerDied","Data":"ca8b59746419f1c0f8f542dd29b95a0384625518d2cb2bf908f14ca8be92e779"}
Jul 11 03:59:59.518489 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.518471    2582 scope.go:117] "RemoveContainer" containerID="72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01"
Jul 11 03:59:59.521309 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.521289    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8j4\" (UniqueName: \"kubernetes.io/projected/c62c474b-fc0f-4db3-a584-a8b6a705fb65-kube-api-access-lt8j4\") pod \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") "
Jul 11 03:59:59.521429 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.521415    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c62c474b-fc0f-4db3-a584-a8b6a705fb65-extensions-socket-volume\") pod \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\" (UID: \"c62c474b-fc0f-4db3-a584-a8b6a705fb65\") "
Jul 11 03:59:59.521938 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.521846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c474b-fc0f-4db3-a584-a8b6a705fb65-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c62c474b-fc0f-4db3-a584-a8b6a705fb65" (UID: "c62c474b-fc0f-4db3-a584-a8b6a705fb65"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 03:59:59.523399 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.523381    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62c474b-fc0f-4db3-a584-a8b6a705fb65-kube-api-access-lt8j4" (OuterVolumeSpecName: "kube-api-access-lt8j4") pod "c62c474b-fc0f-4db3-a584-a8b6a705fb65" (UID: "c62c474b-fc0f-4db3-a584-a8b6a705fb65"). InnerVolumeSpecName "kube-api-access-lt8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 03:59:59.535065 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.535043    2582 scope.go:117] "RemoveContainer" containerID="72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01"
Jul 11 03:59:59.535388 ip-10-0-140-230 kubenswrapper[2582]: E0711 03:59:59.535369    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01\": container with ID starting with 72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01 not found: ID does not exist" containerID="72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01"
Jul 11 03:59:59.535450 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.535398    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01"} err="failed to get container status \"72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01\": rpc error: code = NotFound desc = could not find container \"72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01\": container with ID starting with 72dbb0697aa053f19ef5ca29cc65966c20898a25da9803edf42a25f6b09dbc01 not found: ID does not exist"
Jul 11 03:59:59.622922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.622886    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8j4\" (UniqueName: \"kubernetes.io/projected/c62c474b-fc0f-4db3-a584-a8b6a705fb65-kube-api-access-lt8j4\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:59:59.622922 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.622919    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c62c474b-fc0f-4db3-a584-a8b6a705fb65-extensions-socket-volume\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 03:59:59.834567 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.834489    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"]
Jul 11 03:59:59.837469 ip-10-0-140-230 kubenswrapper[2582]: I0711 03:59:59.837445    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mdcwt"]
Jul 11 04:00:00.142550 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.142473    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"]
Jul 11 04:00:00.142839 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.142827    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62c474b-fc0f-4db3-a584-a8b6a705fb65" containerName="manager"
Jul 11 04:00:00.142881 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.142840    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62c474b-fc0f-4db3-a584-a8b6a705fb65" containerName="manager"
Jul 11 04:00:00.142921 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.142912    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62c474b-fc0f-4db3-a584-a8b6a705fb65" containerName="manager"
Jul 11 04:00:00.147017 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.146996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"
Jul 11 04:00:00.149818 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.149793    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\""
Jul 11 04:00:00.152582 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.152561    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"]
Jul 11 04:00:00.158915 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.158892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"
Jul 11 04:00:00.290466 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.290444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"]
Jul 11 04:00:00.292716 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:00:00.292693    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afa7bcd_7cd2_40a3_8960_c1b9b8c5756f.slice/crio-7cdfa903856507000c7103eee0b2c9916ba8720ac7f446c307cf719d84184280 WatchSource:0}: Error finding container 7cdfa903856507000c7103eee0b2c9916ba8720ac7f446c307cf719d84184280: Status 404 returned error can't find the container with id 7cdfa903856507000c7103eee0b2c9916ba8720ac7f446c307cf719d84184280
Jul 11 04:00:00.294781 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.294761    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 04:00:00.524502 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:00.524465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4" event={"ID":"8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f","Type":"ContainerStarted","Data":"7cdfa903856507000c7103eee0b2c9916ba8720ac7f446c307cf719d84184280"}
Jul 11 04:00:01.648291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:01.648252    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62c474b-fc0f-4db3-a584-a8b6a705fb65" path="/var/lib/kubelet/pods/c62c474b-fc0f-4db3-a584-a8b6a705fb65/volumes"
Jul 11 04:00:03.542528 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:03.542499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4" event={"ID":"8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f","Type":"ContainerStarted","Data":"268d3066f669fdeb12ca6be5ffdfe32fd2de3d671b759b6cb69694dc25037fbb"}
Jul 11 04:00:03.558701 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:03.558655    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4" podStartSLOduration=0.997218383 podStartE2EDuration="3.558636943s" totalImagesPullingTime="2.56141856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:00:03.556643156 +0000 UTC m=+1048.426089721" watchObservedRunningTime="2026-07-11 04:00:03.558636943 +0000 UTC m=+1048.428083506"
Jul 11 04:00:04.548565 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:04.548529    2582 generic.go:356] "Generic (PLEG): container finished" podID="8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f" containerID="268d3066f669fdeb12ca6be5ffdfe32fd2de3d671b759b6cb69694dc25037fbb" exitCode=0
Jul 11 04:00:04.548960 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:04.548603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4" event={"ID":"8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f","Type":"ContainerDied","Data":"268d3066f669fdeb12ca6be5ffdfe32fd2de3d671b759b6cb69694dc25037fbb"}
Jul 11 04:00:05.695049 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:05.695020    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"
Jul 11 04:00:06.559883 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:06.559852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4" event={"ID":"8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f","Type":"ContainerDied","Data":"7cdfa903856507000c7103eee0b2c9916ba8720ac7f446c307cf719d84184280"}
Jul 11 04:00:06.559883 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:06.559876    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729040-qtkd4"
Jul 11 04:00:06.560129 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:06.559900    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cdfa903856507000c7103eee0b2c9916ba8720ac7f446c307cf719d84184280"
Jul 11 04:00:23.861411 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:23.861377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:00:25.042805 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:25.042774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:00:56.814216 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.814179    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"]
Jul 11 04:00:56.814630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.814564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f" containerName="cleanup"
Jul 11 04:00:56.814630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.814575    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f" containerName="cleanup"
Jul 11 04:00:56.814705 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.814654    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f" containerName="cleanup"
Jul 11 04:00:56.822021 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.821995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:56.824510 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.824488    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2468z\""
Jul 11 04:00:56.831555 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.831527    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"]
Jul 11 04:00:56.921907 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.921860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f467436-7dfc-445b-b0ec-41781e1f5047-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7vzfz\" (UID: \"0f467436-7dfc-445b-b0ec-41781e1f5047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:56.922134 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:56.921928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbc5r\" (UniqueName: \"kubernetes.io/projected/0f467436-7dfc-445b-b0ec-41781e1f5047-kube-api-access-wbc5r\") pod \"kuadrant-operator-controller-manager-86878c769f-7vzfz\" (UID: \"0f467436-7dfc-445b-b0ec-41781e1f5047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.023214 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.023176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f467436-7dfc-445b-b0ec-41781e1f5047-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7vzfz\" (UID: \"0f467436-7dfc-445b-b0ec-41781e1f5047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.023410 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.023238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbc5r\" (UniqueName: \"kubernetes.io/projected/0f467436-7dfc-445b-b0ec-41781e1f5047-kube-api-access-wbc5r\") pod \"kuadrant-operator-controller-manager-86878c769f-7vzfz\" (UID: \"0f467436-7dfc-445b-b0ec-41781e1f5047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.023622 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.023601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f467436-7dfc-445b-b0ec-41781e1f5047-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7vzfz\" (UID: \"0f467436-7dfc-445b-b0ec-41781e1f5047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.034121 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.034060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbc5r\" (UniqueName: \"kubernetes.io/projected/0f467436-7dfc-445b-b0ec-41781e1f5047-kube-api-access-wbc5r\") pod \"kuadrant-operator-controller-manager-86878c769f-7vzfz\" (UID: \"0f467436-7dfc-445b-b0ec-41781e1f5047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.133141 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.133021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.275522 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.275493    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"]
Jul 11 04:00:57.278281 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:00:57.278254    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f467436_7dfc_445b_b0ec_41781e1f5047.slice/crio-754cfef48f173c04708cf55cdf4b9f323bd770249226d4db687c95051b8c7252 WatchSource:0}: Error finding container 754cfef48f173c04708cf55cdf4b9f323bd770249226d4db687c95051b8c7252: Status 404 returned error can't find the container with id 754cfef48f173c04708cf55cdf4b9f323bd770249226d4db687c95051b8c7252
Jul 11 04:00:57.801302 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.801266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz" event={"ID":"0f467436-7dfc-445b-b0ec-41781e1f5047","Type":"ContainerStarted","Data":"8bde524d41c763f755b4768064227bb14007e168152b114ab8b00b6e63c1a1e9"}
Jul 11 04:00:57.801480 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.801318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz" event={"ID":"0f467436-7dfc-445b-b0ec-41781e1f5047","Type":"ContainerStarted","Data":"754cfef48f173c04708cf55cdf4b9f323bd770249226d4db687c95051b8c7252"}
Jul 11 04:00:57.801480 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.801431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:00:57.820491 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:00:57.820448    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz" podStartSLOduration=1.820432356 podStartE2EDuration="1.820432356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:00:57.818471728 +0000 UTC m=+1102.687918292" watchObservedRunningTime="2026-07-11 04:00:57.820432356 +0000 UTC m=+1102.689878956"
Jul 11 04:01:08.808346 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:01:08.808320    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7vzfz"
Jul 11 04:01:19.903959 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:01:19.903921    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:01:29.003358 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:01:29.003329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:01:42.105748 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:01:42.105718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:01:53.510968 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:01:53.510931    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:02:05.208037 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:05.207963    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:02:14.808622 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:14.808590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:02:24.305367 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:24.305327    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:02:35.505064 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:35.505026    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:02:35.781415 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:35.781331    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:02:35.786015 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:35.785993    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:02:46.612041 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:46.612007    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:02:55.210497 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:02:55.210461    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:03:06.212179 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:03:06.212144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:03:20.591688 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:03:20.591644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:03:29.805883 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:03:29.805799    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:03:37.304565 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:03:37.304530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:03:47.706386 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:03:47.706351    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:03:57.411116 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:03:57.411058    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:04:06.609568 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:06.609536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:04:18.815405 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:18.815367    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:04:30.611078 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:30.611044    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:04:40.307331 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:40.307301    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:04:47.811963 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:47.811923    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:04:50.688154 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:50.688090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:04:50.741302 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:50.741265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:04:58.306495 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:04:58.306410    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:05:07.802677 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:05:07.802646    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:05:21.503089 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:05:21.503053    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:05:30.815306 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:05:30.815272    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:05:40.709736 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:05:40.709699    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:05:50.006874 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:05:50.006837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:06:03.402330 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:03.402282    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:06:11.309946 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:11.309913    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:06:23.908706 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:23.908671    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:06:35.011035 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:35.010951    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:06:44.099434 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:44.099390    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:06:47.010820 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:47.010783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:06:55.349994 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:55.349949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:06:58.200897 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:06:58.200862    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:07:02.690385 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:02.690353    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:07.711828 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:07.711791    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:07:07.779020 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:07.778987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:10.710480 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:10.710445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:10.775220 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:10.775186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:11.172257 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:11.172222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:16.767276 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:16.767232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:16.821459 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:16.821429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:16.910358 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:16.910321    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:07:16.985424 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:16.985392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:17.085818 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:17.085740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.071440 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.071394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.149568 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.149538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.150317 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.150287    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"]
Jul 11 04:07:18.154216 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.154191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.156774 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.156747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-af4c0929-serving-cert\""
Jul 11 04:07:18.164304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.164278    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"]
Jul 11 04:07:18.218484 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.218457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.245490 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.245457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c765015b-4f6a-4ec9-b669-e100f83ee15a-openshift-service-ca\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.245490 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.245492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpx87\" (UniqueName: \"kubernetes.io/projected/c765015b-4f6a-4ec9-b669-e100f83ee15a-kube-api-access-wpx87\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.245716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.245514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.347089 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.346993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c765015b-4f6a-4ec9-b669-e100f83ee15a-openshift-service-ca\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.347089 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.347032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpx87\" (UniqueName: \"kubernetes.io/projected/c765015b-4f6a-4ec9-b669-e100f83ee15a-kube-api-access-wpx87\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.347089 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.347058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.347363 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:07:18.347264    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-serving-cert: secret "maas-api-e2e-ait-af4c0929-serving-cert" not found
Jul 11 04:07:18.347363 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:07:18.347344    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls podName:c765015b-4f6a-4ec9-b669-e100f83ee15a nodeName:}" failed. No retries permitted until 2026-07-11 04:07:18.847322888 +0000 UTC m=+1483.716769439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls") pod "maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" (UID: "c765015b-4f6a-4ec9-b669-e100f83ee15a") : secret "maas-api-e2e-ait-af4c0929-serving-cert" not found
Jul 11 04:07:18.347756 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.347735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c765015b-4f6a-4ec9-b669-e100f83ee15a-openshift-service-ca\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.358425 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.358392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpx87\" (UniqueName: \"kubernetes.io/projected/c765015b-4f6a-4ec9-b669-e100f83ee15a-kube-api-access-wpx87\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.474304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.474277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.718830 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.718736    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.819973 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.819928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:18.852912 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.852874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:18.855512 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:18.855492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls\") pod \"maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:19.068870 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:19.068834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:19.220994 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:19.220969    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"]
Jul 11 04:07:19.222850 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:07:19.222824    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc765015b_4f6a_4ec9_b669_e100f83ee15a.slice/crio-07b27b7b38b8c92aa3cc14b04571350ee4cebe44c88f35183a137e59264b1bc1 WatchSource:0}: Error finding container 07b27b7b38b8c92aa3cc14b04571350ee4cebe44c88f35183a137e59264b1bc1: Status 404 returned error can't find the container with id 07b27b7b38b8c92aa3cc14b04571350ee4cebe44c88f35183a137e59264b1bc1
Jul 11 04:07:19.224316 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:19.224302    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 04:07:19.664590 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:19.664554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" event={"ID":"c765015b-4f6a-4ec9-b669-e100f83ee15a","Type":"ContainerStarted","Data":"07b27b7b38b8c92aa3cc14b04571350ee4cebe44c88f35183a137e59264b1bc1"}
Jul 11 04:07:20.671968 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:20.671931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" event={"ID":"c765015b-4f6a-4ec9-b669-e100f83ee15a","Type":"ContainerStarted","Data":"4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50"}
Jul 11 04:07:20.672387 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:20.671993    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:20.687748 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:20.687702    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" podStartSLOduration=2.23376364 podStartE2EDuration="2.687682425s" totalImagesPullingTime="453.918785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:07:20.687064871 +0000 UTC m=+1485.556511442" watchObservedRunningTime="2026-07-11 04:07:20.687682425 +0000 UTC m=+1485.557128988"
Jul 11 04:07:20.902808 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:20.902777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:26.682460 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:26.682428    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:28.504744 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:28.504708    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:07:33.475674 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.475630    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"]
Jul 11 04:07:33.476239 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.475859    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" podUID="c765015b-4f6a-4ec9-b669-e100f83ee15a" containerName="maas-api" containerID="cri-o://4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50" gracePeriod=30
Jul 11 04:07:33.502265 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.502236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:33.742495 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.742472    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:33.744613 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.744591    2582 generic.go:356] "Generic (PLEG): container finished" podID="c765015b-4f6a-4ec9-b669-e100f83ee15a" containerID="4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50" exitCode=0
Jul 11 04:07:33.744720 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.744626    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"
Jul 11 04:07:33.744720 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.744673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" event={"ID":"c765015b-4f6a-4ec9-b669-e100f83ee15a","Type":"ContainerDied","Data":"4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50"}
Jul 11 04:07:33.744799 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.744729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn" event={"ID":"c765015b-4f6a-4ec9-b669-e100f83ee15a","Type":"ContainerDied","Data":"07b27b7b38b8c92aa3cc14b04571350ee4cebe44c88f35183a137e59264b1bc1"}
Jul 11 04:07:33.744799 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.744757    2582 scope.go:117] "RemoveContainer" containerID="4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50"
Jul 11 04:07:33.754271 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.754249    2582 scope.go:117] "RemoveContainer" containerID="4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50"
Jul 11 04:07:33.754626 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:07:33.754599    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50\": container with ID starting with 4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50 not found: ID does not exist" containerID="4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50"
Jul 11 04:07:33.754711 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.754639    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50"} err="failed to get container status \"4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50\": rpc error: code = NotFound desc = could not find container \"4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50\": container with ID starting with 4270c748e9026d95c52733362ea4322b8137ba922eb89febdddd6285bc61dd50 not found: ID does not exist"
Jul 11 04:07:33.774779 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.774755    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpx87\" (UniqueName: \"kubernetes.io/projected/c765015b-4f6a-4ec9-b669-e100f83ee15a-kube-api-access-wpx87\") pod \"c765015b-4f6a-4ec9-b669-e100f83ee15a\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") "
Jul 11 04:07:33.774976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.774788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls\") pod \"c765015b-4f6a-4ec9-b669-e100f83ee15a\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") "
Jul 11 04:07:33.774976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.774880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c765015b-4f6a-4ec9-b669-e100f83ee15a-openshift-service-ca\") pod \"c765015b-4f6a-4ec9-b669-e100f83ee15a\" (UID: \"c765015b-4f6a-4ec9-b669-e100f83ee15a\") "
Jul 11 04:07:33.775286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.775257    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c765015b-4f6a-4ec9-b669-e100f83ee15a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c765015b-4f6a-4ec9-b669-e100f83ee15a" (UID: "c765015b-4f6a-4ec9-b669-e100f83ee15a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:07:33.777210 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.777181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c765015b-4f6a-4ec9-b669-e100f83ee15a-kube-api-access-wpx87" (OuterVolumeSpecName: "kube-api-access-wpx87") pod "c765015b-4f6a-4ec9-b669-e100f83ee15a" (UID: "c765015b-4f6a-4ec9-b669-e100f83ee15a"). InnerVolumeSpecName "kube-api-access-wpx87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:07:33.777210 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.777198    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c765015b-4f6a-4ec9-b669-e100f83ee15a" (UID: "c765015b-4f6a-4ec9-b669-e100f83ee15a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:07:33.876471 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.876440    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpx87\" (UniqueName: \"kubernetes.io/projected/c765015b-4f6a-4ec9-b669-e100f83ee15a-kube-api-access-wpx87\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:07:33.876471 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.876469    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c765015b-4f6a-4ec9-b669-e100f83ee15a-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:07:33.876659 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:33.876481    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c765015b-4f6a-4ec9-b669-e100f83ee15a-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:07:34.067106 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:34.067073    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"]
Jul 11 04:07:34.070995 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:34.070970    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af4c0929-68b97cf649-mrtsn"]
Jul 11 04:07:35.646799 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:35.646768    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c765015b-4f6a-4ec9-b669-e100f83ee15a" path="/var/lib/kubelet/pods/c765015b-4f6a-4ec9-b669-e100f83ee15a/volumes"
Jul 11 04:07:35.808821 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:35.808795    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:07:35.815843 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:35.815819    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:07:37.098615 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:37.098581    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:07:38.181173 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:38.181135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:43.705630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:43.705595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:43.768571 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:43.768521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:44.156539 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:44.156507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:48.373286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:48.373246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:48.755281 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:48.755243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:48.858966 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:48.858929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:48.957728 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:48.957698    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:50.179653 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.179622    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:50.404423 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.404394    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"]
Jul 11 04:07:50.404759 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.404746    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c765015b-4f6a-4ec9-b669-e100f83ee15a" containerName="maas-api"
Jul 11 04:07:50.404808 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.404760    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c765015b-4f6a-4ec9-b669-e100f83ee15a" containerName="maas-api"
Jul 11 04:07:50.404857 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.404849    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c765015b-4f6a-4ec9-b669-e100f83ee15a" containerName="maas-api"
Jul 11 04:07:50.409512 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.409495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.411668 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.411645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4b08374c-serving-cert\""
Jul 11 04:07:50.419339 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.419311    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"]
Jul 11 04:07:50.437857 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.437757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:50.508751 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.508721    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:50.525173 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.525137    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.525334 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.525192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-maas-api-tls\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.525334 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.525260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxc5\" (UniqueName: \"kubernetes.io/projected/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-kube-api-access-ctxc5\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.626702 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.626656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.626916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.626817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-maas-api-tls\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.626916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.626860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxc5\" (UniqueName: \"kubernetes.io/projected/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-kube-api-access-ctxc5\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.627487 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.627460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.629438 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.629409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-maas-api-tls\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.634655 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.634632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxc5\" (UniqueName: \"kubernetes.io/projected/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-kube-api-access-ctxc5\") pod \"maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.722320 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.722229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:50.862798 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.862760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:50.869720 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.869388    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"]
Jul 11 04:07:50.874487 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:07:50.874457    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d39ad8f_438b_4451_98e1_020dc3cb5f9c.slice/crio-f62b0f060e9592da27e5434d77f8af32e0394724ee187b48960fd14bd324384e WatchSource:0}: Error finding container f62b0f060e9592da27e5434d77f8af32e0394724ee187b48960fd14bd324384e: Status 404 returned error can't find the container with id f62b0f060e9592da27e5434d77f8af32e0394724ee187b48960fd14bd324384e
Jul 11 04:07:50.962262 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:50.962229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:51.837650 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:51.837608    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs" event={"ID":"4d39ad8f-438b-4451-98e1-020dc3cb5f9c","Type":"ContainerStarted","Data":"b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00"}
Jul 11 04:07:51.838126 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:51.837667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs" event={"ID":"4d39ad8f-438b-4451-98e1-020dc3cb5f9c","Type":"ContainerStarted","Data":"f62b0f060e9592da27e5434d77f8af32e0394724ee187b48960fd14bd324384e"}
Jul 11 04:07:51.838126 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:51.837742    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:51.854410 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:51.854363    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs" podStartSLOduration=1.48864389 podStartE2EDuration="1.854346253s" totalImagesPullingTime="365.702363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:07:51.852684024 +0000 UTC m=+1516.722130589" watchObservedRunningTime="2026-07-11 04:07:51.854346253 +0000 UTC m=+1516.723792817"
Jul 11 04:07:54.765319 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:54.765289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:54.888714 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:54.888685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:07:57.208909 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:57.208824    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:07:57.849278 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:57.849252    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:07:57.998363 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:07:57.998329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:08:54.864135 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:54.864079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:08:54.972026 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:54.971996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:08:55.295642 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.295603    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"]
Jul 11 04:08:55.302633 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.302546    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs" podUID="4d39ad8f-438b-4451-98e1-020dc3cb5f9c" containerName="maas-api" containerID="cri-o://b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00" gracePeriod=30
Jul 11 04:08:55.316170 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.316145    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:08:55.562058 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.562033    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:08:55.697298 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.697266    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-maas-api-tls\") pod \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") "
Jul 11 04:08:55.697492 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.697323    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctxc5\" (UniqueName: \"kubernetes.io/projected/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-kube-api-access-ctxc5\") pod \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") "
Jul 11 04:08:55.697492 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.697439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-openshift-service-ca\") pod \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\" (UID: \"4d39ad8f-438b-4451-98e1-020dc3cb5f9c\") "
Jul 11 04:08:55.697817 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.697784    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d39ad8f-438b-4451-98e1-020dc3cb5f9c" (UID: "4d39ad8f-438b-4451-98e1-020dc3cb5f9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:08:55.699641 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.699615    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-kube-api-access-ctxc5" (OuterVolumeSpecName: "kube-api-access-ctxc5") pod "4d39ad8f-438b-4451-98e1-020dc3cb5f9c" (UID: "4d39ad8f-438b-4451-98e1-020dc3cb5f9c"). InnerVolumeSpecName "kube-api-access-ctxc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:08:55.699641 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.699634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d39ad8f-438b-4451-98e1-020dc3cb5f9c" (UID: "4d39ad8f-438b-4451-98e1-020dc3cb5f9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:08:55.798809 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.798733    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxc5\" (UniqueName: \"kubernetes.io/projected/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-kube-api-access-ctxc5\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:08:55.798809 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.798760    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:08:55.798809 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:55.798770    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d39ad8f-438b-4451-98e1-020dc3cb5f9c-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:08:56.155638 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.155538    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d39ad8f-438b-4451-98e1-020dc3cb5f9c" containerID="b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00" exitCode=0
Jul 11 04:08:56.155638 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.155591    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"
Jul 11 04:08:56.156197 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.155630    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs" event={"ID":"4d39ad8f-438b-4451-98e1-020dc3cb5f9c","Type":"ContainerDied","Data":"b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00"}
Jul 11 04:08:56.156197 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.155696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs" event={"ID":"4d39ad8f-438b-4451-98e1-020dc3cb5f9c","Type":"ContainerDied","Data":"f62b0f060e9592da27e5434d77f8af32e0394724ee187b48960fd14bd324384e"}
Jul 11 04:08:56.156197 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.155730    2582 scope.go:117] "RemoveContainer" containerID="b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00"
Jul 11 04:08:56.165084 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.165069    2582 scope.go:117] "RemoveContainer" containerID="b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00"
Jul 11 04:08:56.165338 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:08:56.165316    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00\": container with ID starting with b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00 not found: ID does not exist" containerID="b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00"
Jul 11 04:08:56.165382 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.165347    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00"} err="failed to get container status \"b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00\": rpc error: code = NotFound desc = could not find container \"b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00\": container with ID starting with b30a56c3c16c88dfcc4ac3e3e791756b9f76850dd529840ef293d05c36d55b00 not found: ID does not exist"
Jul 11 04:08:56.176564 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.176543    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"]
Jul 11 04:08:56.180509 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:56.180484    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b08374c-7f8c9dd7bf-4qbxs"]
Jul 11 04:08:57.400822 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:57.400790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:08:57.646406 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:08:57.646374    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d39ad8f-438b-4451-98e1-020dc3cb5f9c" path="/var/lib/kubelet/pods/4d39ad8f-438b-4451-98e1-020dc3cb5f9c/volumes"
Jul 11 04:09:00.884638 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:00.884563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:00.950872 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:00.950842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:01.083289 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:01.083244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:01.419792 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:01.419757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:01.475384 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:01.475314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:02.182698 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.182666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:02.288476 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.288423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:02.294229 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.294200    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"]
Jul 11 04:09:02.294612 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.294598    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d39ad8f-438b-4451-98e1-020dc3cb5f9c" containerName="maas-api"
Jul 11 04:09:02.294678 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.294613    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d39ad8f-438b-4451-98e1-020dc3cb5f9c" containerName="maas-api"
Jul 11 04:09:02.294735 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.294683    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d39ad8f-438b-4451-98e1-020dc3cb5f9c" containerName="maas-api"
Jul 11 04:09:02.299389 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.299368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.301604 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.301588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c784f07-serving-cert\""
Jul 11 04:09:02.306536 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.306513    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"]
Jul 11 04:09:02.383083 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.383034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:02.456530 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.456439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8kh\" (UniqueName: \"kubernetes.io/projected/5232a72a-5ea3-4015-9121-f38415857150-kube-api-access-rp8kh\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.456530 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.456479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.456760 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.456562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5232a72a-5ea3-4015-9121-f38415857150-openshift-service-ca\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.557204 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.557174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5232a72a-5ea3-4015-9121-f38415857150-openshift-service-ca\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.557401 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.557241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8kh\" (UniqueName: \"kubernetes.io/projected/5232a72a-5ea3-4015-9121-f38415857150-kube-api-access-rp8kh\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.557401 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.557276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.557516 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:09:02.557405    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-serving-cert: secret "maas-api-e2e-ait-1c784f07-serving-cert" not found
Jul 11 04:09:02.557516 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:09:02.557471    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls podName:5232a72a-5ea3-4015-9121-f38415857150 nodeName:}" failed. No retries permitted until 2026-07-11 04:09:03.057451442 +0000 UTC m=+1587.926897985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls") pod "maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" (UID: "5232a72a-5ea3-4015-9121-f38415857150") : secret "maas-api-e2e-ait-1c784f07-serving-cert" not found
Jul 11 04:09:02.558008 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.557983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5232a72a-5ea3-4015-9121-f38415857150-openshift-service-ca\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.567542 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.567515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8kh\" (UniqueName: \"kubernetes.io/projected/5232a72a-5ea3-4015-9121-f38415857150-kube-api-access-rp8kh\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:02.916046 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:02.915997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:03.016254 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:03.016208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:03.063868 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:03.063833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:03.064041 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:09:03.063998    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-serving-cert: secret "maas-api-e2e-ait-1c784f07-serving-cert" not found
Jul 11 04:09:03.064115 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:09:03.064074    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls podName:5232a72a-5ea3-4015-9121-f38415857150 nodeName:}" failed. No retries permitted until 2026-07-11 04:09:04.064054076 +0000 UTC m=+1588.933500638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls") pod "maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" (UID: "5232a72a-5ea3-4015-9121-f38415857150") : secret "maas-api-e2e-ait-1c784f07-serving-cert" not found
Jul 11 04:09:03.226263 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:03.226167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:04.073113 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:04.073071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:04.075738 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:04.075712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") pod \"maas-api-e2e-ait-1c784f07-64b66dbc7-745lj\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:04.111723 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:04.111692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:04.253255 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:04.253229    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"]
Jul 11 04:09:04.256136 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:09:04.255645    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5232a72a_5ea3_4015_9121_f38415857150.slice/crio-ca1fbc85ffb5f0e0dce361489e50c0ce9a9679e39c38b6d03e55b0237d2aa694 WatchSource:0}: Error finding container ca1fbc85ffb5f0e0dce361489e50c0ce9a9679e39c38b6d03e55b0237d2aa694: Status 404 returned error can't find the container with id ca1fbc85ffb5f0e0dce361489e50c0ce9a9679e39c38b6d03e55b0237d2aa694
Jul 11 04:09:04.683510 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:04.683479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:09:05.206922 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:05.206876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" event={"ID":"5232a72a-5ea3-4015-9121-f38415857150","Type":"ContainerStarted","Data":"8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1"}
Jul 11 04:09:05.207144 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:05.206990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" event={"ID":"5232a72a-5ea3-4015-9121-f38415857150","Type":"ContainerStarted","Data":"ca1fbc85ffb5f0e0dce361489e50c0ce9a9679e39c38b6d03e55b0237d2aa694"}
Jul 11 04:09:05.207144 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:05.207021    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:09:05.224423 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:05.224368    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" podStartSLOduration=2.879899257 podStartE2EDuration="3.224352952s" totalImagesPullingTime="344.453695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:09:05.222254587 +0000 UTC m=+1590.091701154" watchObservedRunningTime="2026-07-11 04:09:05.224352952 +0000 UTC m=+1590.093799516"
Jul 11 04:09:08.711075 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:08.711044    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:09:11.218637 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:09:11.218606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:10:17.699235 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:17.699203    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"]
Jul 11 04:10:17.699871 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:17.699426    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" podUID="5232a72a-5ea3-4015-9121-f38415857150" containerName="maas-api" containerID="cri-o://8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1" gracePeriod=30
Jul 11 04:10:17.722344 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:17.722320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:17.962789 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:17.962729    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:10:18.014291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.014260    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") pod \"5232a72a-5ea3-4015-9121-f38415857150\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") "
Jul 11 04:10:18.014465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.014327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp8kh\" (UniqueName: \"kubernetes.io/projected/5232a72a-5ea3-4015-9121-f38415857150-kube-api-access-rp8kh\") pod \"5232a72a-5ea3-4015-9121-f38415857150\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") "
Jul 11 04:10:18.014465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.014375    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5232a72a-5ea3-4015-9121-f38415857150-openshift-service-ca\") pod \"5232a72a-5ea3-4015-9121-f38415857150\" (UID: \"5232a72a-5ea3-4015-9121-f38415857150\") "
Jul 11 04:10:18.014763 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.014741    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5232a72a-5ea3-4015-9121-f38415857150-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5232a72a-5ea3-4015-9121-f38415857150" (UID: "5232a72a-5ea3-4015-9121-f38415857150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:10:18.016407 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.016385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5232a72a-5ea3-4015-9121-f38415857150" (UID: "5232a72a-5ea3-4015-9121-f38415857150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:10:18.016498 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.016481    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5232a72a-5ea3-4015-9121-f38415857150-kube-api-access-rp8kh" (OuterVolumeSpecName: "kube-api-access-rp8kh") pod "5232a72a-5ea3-4015-9121-f38415857150" (UID: "5232a72a-5ea3-4015-9121-f38415857150"). InnerVolumeSpecName "kube-api-access-rp8kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:10:18.115906 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.115876    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8kh\" (UniqueName: \"kubernetes.io/projected/5232a72a-5ea3-4015-9121-f38415857150-kube-api-access-rp8kh\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:10:18.115906 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.115906    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5232a72a-5ea3-4015-9121-f38415857150-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:10:18.116127 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.115917    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5232a72a-5ea3-4015-9121-f38415857150-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:10:18.572931 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.572892    2582 generic.go:356] "Generic (PLEG): container finished" podID="5232a72a-5ea3-4015-9121-f38415857150" containerID="8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1" exitCode=0
Jul 11 04:10:18.573131 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.572955    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"
Jul 11 04:10:18.573131 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.572984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" event={"ID":"5232a72a-5ea3-4015-9121-f38415857150","Type":"ContainerDied","Data":"8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1"}
Jul 11 04:10:18.573131 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.573061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj" event={"ID":"5232a72a-5ea3-4015-9121-f38415857150","Type":"ContainerDied","Data":"ca1fbc85ffb5f0e0dce361489e50c0ce9a9679e39c38b6d03e55b0237d2aa694"}
Jul 11 04:10:18.573131 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.573083    2582 scope.go:117] "RemoveContainer" containerID="8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1"
Jul 11 04:10:18.583308 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.583288    2582 scope.go:117] "RemoveContainer" containerID="8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1"
Jul 11 04:10:18.583581 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:10:18.583561    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1\": container with ID starting with 8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1 not found: ID does not exist" containerID="8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1"
Jul 11 04:10:18.583636 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.583589    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1"} err="failed to get container status \"8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1\": rpc error: code = NotFound desc = could not find container \"8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1\": container with ID starting with 8c2d1d9151982a8e2e2cc2abfa40713e5b4dd98ce8c9398526b1109a8b8a5be1 not found: ID does not exist"
Jul 11 04:10:18.593065 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.593040    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"]
Jul 11 04:10:18.595530 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:18.595507    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c784f07-64b66dbc7-745lj"]
Jul 11 04:10:19.313073 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:19.313034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:10:19.645541 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:19.645464    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5232a72a-5ea3-4015-9121-f38415857150" path="/var/lib/kubelet/pods/5232a72a-5ea3-4015-9121-f38415857150/volumes"
Jul 11 04:10:19.798291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:19.798255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:19.848359 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:19.848326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:22.415458 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:22.415425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:28.048983 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:28.048948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:28.149672 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:28.149640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:28.796228 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:28.796197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:28.896927 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:28.896894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:33.443625 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:33.443588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:33.751680 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:33.751644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:34.857300 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.857261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:34.932291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.932265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:34.936046 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.936012    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"]
Jul 11 04:10:34.936556 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.936533    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5232a72a-5ea3-4015-9121-f38415857150" containerName="maas-api"
Jul 11 04:10:34.936556 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.936557    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5232a72a-5ea3-4015-9121-f38415857150" containerName="maas-api"
Jul 11 04:10:34.936749 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.936679    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5232a72a-5ea3-4015-9121-f38415857150" containerName="maas-api"
Jul 11 04:10:34.939630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.939614    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:34.942256 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.942236    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-55671289-serving-cert\""
Jul 11 04:10:34.952202 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.952179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"]
Jul 11 04:10:34.999746 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:34.999713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:35.075089 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.075050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.075303 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.075179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntgw\" (UniqueName: \"kubernetes.io/projected/48516f5d-00cf-4af3-9656-8d1d218def6d-kube-api-access-tntgw\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.075303 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.075226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48516f5d-00cf-4af3-9656-8d1d218def6d-openshift-service-ca\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.176458 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.176375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntgw\" (UniqueName: \"kubernetes.io/projected/48516f5d-00cf-4af3-9656-8d1d218def6d-kube-api-access-tntgw\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.176458 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.176413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48516f5d-00cf-4af3-9656-8d1d218def6d-openshift-service-ca\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.176638 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.176471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.176638 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:10:35.176594    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-55671289-serving-cert: secret "maas-api-e2e-derive-55671289-serving-cert" not found
Jul 11 04:10:35.176732 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:10:35.176663    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls podName:48516f5d-00cf-4af3-9656-8d1d218def6d nodeName:}" failed. No retries permitted until 2026-07-11 04:10:35.676643385 +0000 UTC m=+1680.546089927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls") pod "maas-api-e2e-derive-55671289-55d8585f95-dng6f" (UID: "48516f5d-00cf-4af3-9656-8d1d218def6d") : secret "maas-api-e2e-derive-55671289-serving-cert" not found
Jul 11 04:10:35.177090 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.177069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48516f5d-00cf-4af3-9656-8d1d218def6d-openshift-service-ca\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.184619 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.184598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntgw\" (UniqueName: \"kubernetes.io/projected/48516f5d-00cf-4af3-9656-8d1d218def6d-kube-api-access-tntgw\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.492431 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.492403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:35.592582 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.592547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:35.679769 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.679740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.682361 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.682337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls\") pod \"maas-api-e2e-derive-55671289-55d8585f95-dng6f\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.695262 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.695235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:35.850836 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.850799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:35.993975 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:35.993929    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"]
Jul 11 04:10:35.996958 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:10:35.996924    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48516f5d_00cf_4af3_9656_8d1d218def6d.slice/crio-5bb6b9cb653cdcfce723c07a28f5f7e1dcdd3d3a46cae2fea5abaf459a32858f WatchSource:0}: Error finding container 5bb6b9cb653cdcfce723c07a28f5f7e1dcdd3d3a46cae2fea5abaf459a32858f: Status 404 returned error can't find the container with id 5bb6b9cb653cdcfce723c07a28f5f7e1dcdd3d3a46cae2fea5abaf459a32858f
Jul 11 04:10:36.664708 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:36.664675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f" event={"ID":"48516f5d-00cf-4af3-9656-8d1d218def6d","Type":"ContainerStarted","Data":"975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df"}
Jul 11 04:10:36.664885 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:36.664730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f" event={"ID":"48516f5d-00cf-4af3-9656-8d1d218def6d","Type":"ContainerStarted","Data":"5bb6b9cb653cdcfce723c07a28f5f7e1dcdd3d3a46cae2fea5abaf459a32858f"}
Jul 11 04:10:36.664885 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:36.664841    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:36.702812 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:36.702710    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f" podStartSLOduration=2.259416288 podStartE2EDuration="2.702693585s" totalImagesPullingTime="443.277297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:10:36.69964034 +0000 UTC m=+1681.569086903" watchObservedRunningTime="2026-07-11 04:10:36.702693585 +0000 UTC m=+1681.572140149"
Jul 11 04:10:37.610162 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:37.610126    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:10:42.674393 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:42.674359    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:10:54.729305 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:54.729274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:54.780052 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:54.780019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:10:58.213696 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:10:58.213670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:11:49.653740 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.653707    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"]
Jul 11 04:11:49.656271 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.654056    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f" podUID="48516f5d-00cf-4af3-9656-8d1d218def6d" containerName="maas-api" containerID="cri-o://975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df" gracePeriod=30
Jul 11 04:11:49.679508 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.679476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:11:49.912269 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.912217    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:11:49.951991 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.951940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48516f5d-00cf-4af3-9656-8d1d218def6d-openshift-service-ca\") pod \"48516f5d-00cf-4af3-9656-8d1d218def6d\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") "
Jul 11 04:11:49.952176 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.952021    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tntgw\" (UniqueName: \"kubernetes.io/projected/48516f5d-00cf-4af3-9656-8d1d218def6d-kube-api-access-tntgw\") pod \"48516f5d-00cf-4af3-9656-8d1d218def6d\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") "
Jul 11 04:11:49.952176 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.952131    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls\") pod \"48516f5d-00cf-4af3-9656-8d1d218def6d\" (UID: \"48516f5d-00cf-4af3-9656-8d1d218def6d\") "
Jul 11 04:11:49.952741 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.952707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48516f5d-00cf-4af3-9656-8d1d218def6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48516f5d-00cf-4af3-9656-8d1d218def6d" (UID: "48516f5d-00cf-4af3-9656-8d1d218def6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:11:49.954728 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.954695    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48516f5d-00cf-4af3-9656-8d1d218def6d-kube-api-access-tntgw" (OuterVolumeSpecName: "kube-api-access-tntgw") pod "48516f5d-00cf-4af3-9656-8d1d218def6d" (UID: "48516f5d-00cf-4af3-9656-8d1d218def6d"). InnerVolumeSpecName "kube-api-access-tntgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:11:49.954890 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:49.954871    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48516f5d-00cf-4af3-9656-8d1d218def6d" (UID: "48516f5d-00cf-4af3-9656-8d1d218def6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:11:50.024372 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.024339    2582 generic.go:356] "Generic (PLEG): container finished" podID="48516f5d-00cf-4af3-9656-8d1d218def6d" containerID="975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df" exitCode=0
Jul 11 04:11:50.024566 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.024383    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"
Jul 11 04:11:50.024566 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.024431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f" event={"ID":"48516f5d-00cf-4af3-9656-8d1d218def6d","Type":"ContainerDied","Data":"975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df"}
Jul 11 04:11:50.024566 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.024477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f" event={"ID":"48516f5d-00cf-4af3-9656-8d1d218def6d","Type":"ContainerDied","Data":"5bb6b9cb653cdcfce723c07a28f5f7e1dcdd3d3a46cae2fea5abaf459a32858f"}
Jul 11 04:11:50.024566 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.024499    2582 scope.go:117] "RemoveContainer" containerID="975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df"
Jul 11 04:11:50.034196 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.034173    2582 scope.go:117] "RemoveContainer" containerID="975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df"
Jul 11 04:11:50.034471 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:11:50.034454    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df\": container with ID starting with 975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df not found: ID does not exist" containerID="975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df"
Jul 11 04:11:50.034514 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.034482    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df"} err="failed to get container status \"975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df\": rpc error: code = NotFound desc = could not find container \"975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df\": container with ID starting with 975aac347c63312dc1d57b8024ee16e48d79b290c572d9fbcb6c0e5f0d2c18df not found: ID does not exist"
Jul 11 04:11:50.048001 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.047968    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"]
Jul 11 04:11:50.049734 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.049713    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55671289-55d8585f95-dng6f"]
Jul 11 04:11:50.053055 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.053032    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48516f5d-00cf-4af3-9656-8d1d218def6d-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:11:50.053055 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.053054    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntgw\" (UniqueName: \"kubernetes.io/projected/48516f5d-00cf-4af3-9656-8d1d218def6d-kube-api-access-tntgw\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:11:50.053221 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:50.053067    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48516f5d-00cf-4af3-9656-8d1d218def6d-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:11:51.646071 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:51.646043    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48516f5d-00cf-4af3-9656-8d1d218def6d" path="/var/lib/kubelet/pods/48516f5d-00cf-4af3-9656-8d1d218def6d/volumes"
Jul 11 04:11:52.707947 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:52.707916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:11:54.368726 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:54.368696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:11:59.903736 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:59.903697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:11:59.961758 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:11:59.961722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:04.558681 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:04.558641    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:05.130493 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:05.130456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:05.231294 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:05.231259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:05.330077 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:05.330042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:06.818932 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:06.818891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:11.446464 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:11.446429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:12.437118 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:12.437073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:12.542193 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:12.542159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:12.637646 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:12.637612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:12.738183 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:12.738149    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:13.168900 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:13.168818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:35.837527 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:35.837496    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:12:35.844320 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:35.844296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:12:39.165668 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:39.165632    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:44.816742 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:44.816701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:44.925199 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:44.925162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:12:45.021158 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:12:45.021087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:00.925506 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:00.925469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:06.798373 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:06.798333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:06.898782 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:06.898752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:06.998494 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:06.998457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:17.032820 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:17.032784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:17.133754 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:17.133707    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:17.537505 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:17.537473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:18.343658 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:18.343619    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:24.087547 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:24.087517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:24.189460 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:24.189430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:24.288427 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:24.288397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:24.388329 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:24.388202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:25.199220 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:25.199183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:25.294190 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:25.294160    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:25.394749 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:25.394714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:26.406071 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:26.406034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:13:27.049336 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:27.049301    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:32.683974 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:32.683941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:32.784742 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:32.784715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:33.369067 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:33.369024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:35.009874 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:35.009843    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:13:39.120296 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:39.120258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:39.176240 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:39.176205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:39.362053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:39.362017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:39.689906 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:39.689856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:40.420559 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:40.420516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:42.761431 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.761392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:42.861850 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.861801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:42.904567 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.904539    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"]
Jul 11 04:13:42.904969 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.904953    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48516f5d-00cf-4af3-9656-8d1d218def6d" containerName="maas-api"
Jul 11 04:13:42.905056 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.904972    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48516f5d-00cf-4af3-9656-8d1d218def6d" containerName="maas-api"
Jul 11 04:13:42.905136 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.905065    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48516f5d-00cf-4af3-9656-8d1d218def6d" containerName="maas-api"
Jul 11 04:13:42.908223 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.908205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:42.910454 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.910435    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d7da7c14-serving-cert\""
Jul 11 04:13:42.920310 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.920287    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"]
Jul 11 04:13:42.963345 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:42.963318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:43.078298 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.078188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.078298 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.078273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c2d6-dd2e-46e0-96b4-f104b0132650-openshift-service-ca\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.078521 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.078341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jbgp\" (UniqueName: \"kubernetes.io/projected/8f33c2d6-dd2e-46e0-96b4-f104b0132650-kube-api-access-2jbgp\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.179181 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.179144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jbgp\" (UniqueName: \"kubernetes.io/projected/8f33c2d6-dd2e-46e0-96b4-f104b0132650-kube-api-access-2jbgp\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.179396 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.179223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.179396 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.179254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c2d6-dd2e-46e0-96b4-f104b0132650-openshift-service-ca\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.179520 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:13:43.179387    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-serving-cert: secret "maas-api-e2e-mt-d7da7c14-serving-cert" not found
Jul 11 04:13:43.179520 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:13:43.179480    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls podName:8f33c2d6-dd2e-46e0-96b4-f104b0132650 nodeName:}" failed. No retries permitted until 2026-07-11 04:13:43.679457085 +0000 UTC m=+1868.548903647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls") pod "maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" (UID: "8f33c2d6-dd2e-46e0-96b4-f104b0132650") : secret "maas-api-e2e-mt-d7da7c14-serving-cert" not found
Jul 11 04:13:43.179842 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.179825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c2d6-dd2e-46e0-96b4-f104b0132650-openshift-service-ca\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.189989 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.189965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jbgp\" (UniqueName: \"kubernetes.io/projected/8f33c2d6-dd2e-46e0-96b4-f104b0132650-kube-api-access-2jbgp\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.383999 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.383884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:43.484414 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.484380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:43.584131 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.584079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:43.684044 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.683949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.686633 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.686607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls\") pod \"maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.820787 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.820751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:43.970926 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.970902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"]
Jul 11 04:13:43.972964 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:13:43.972933    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f33c2d6_dd2e_46e0_96b4_f104b0132650.slice/crio-ab38b69a0c9c3af88483231b68548ad298447b7e6a74f812214b9144d2711390 WatchSource:0}: Error finding container ab38b69a0c9c3af88483231b68548ad298447b7e6a74f812214b9144d2711390: Status 404 returned error can't find the container with id ab38b69a0c9c3af88483231b68548ad298447b7e6a74f812214b9144d2711390
Jul 11 04:13:43.974296 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:43.974276    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 04:13:44.596331 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:44.596289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" event={"ID":"8f33c2d6-dd2e-46e0-96b4-f104b0132650","Type":"ContainerStarted","Data":"ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375"}
Jul 11 04:13:44.596505 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:44.596362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" event={"ID":"8f33c2d6-dd2e-46e0-96b4-f104b0132650","Type":"ContainerStarted","Data":"ab38b69a0c9c3af88483231b68548ad298447b7e6a74f812214b9144d2711390"}
Jul 11 04:13:44.596505 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:44.596392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:44.612771 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:44.612724    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" podStartSLOduration=2.138925554 podStartE2EDuration="2.612703081s" totalImagesPullingTime="473.777527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:13:44.612274409 +0000 UTC m=+1869.481720973" watchObservedRunningTime="2026-07-11 04:13:44.612703081 +0000 UTC m=+1869.482149647"
Jul 11 04:13:45.504598 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:45.504566    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:13:47.275946 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:47.275915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:47.949041 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:47.949007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:48.045608 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:48.045579    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:48.383670 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:48.383631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:50.606751 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:50.606723    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:13:54.231377 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:54.231338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:13:54.331472 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:13:54.331444    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:04.938246 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:04.938152    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"]
Jul 11 04:14:04.938708 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:04.938354    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" podUID="8f33c2d6-dd2e-46e0-96b4-f104b0132650" containerName="maas-api" containerID="cri-o://ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375" gracePeriod=30
Jul 11 04:14:04.969582 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:04.969549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:05.197767 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.197703    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:14:05.272125 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.272073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jbgp\" (UniqueName: \"kubernetes.io/projected/8f33c2d6-dd2e-46e0-96b4-f104b0132650-kube-api-access-2jbgp\") pod \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") "
Jul 11 04:14:05.272311 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.272161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c2d6-dd2e-46e0-96b4-f104b0132650-openshift-service-ca\") pod \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") "
Jul 11 04:14:05.272311 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.272223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls\") pod \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\" (UID: \"8f33c2d6-dd2e-46e0-96b4-f104b0132650\") "
Jul 11 04:14:05.272555 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.272530    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f33c2d6-dd2e-46e0-96b4-f104b0132650-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f33c2d6-dd2e-46e0-96b4-f104b0132650" (UID: "8f33c2d6-dd2e-46e0-96b4-f104b0132650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:14:05.274402 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.274376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f33c2d6-dd2e-46e0-96b4-f104b0132650-kube-api-access-2jbgp" (OuterVolumeSpecName: "kube-api-access-2jbgp") pod "8f33c2d6-dd2e-46e0-96b4-f104b0132650" (UID: "8f33c2d6-dd2e-46e0-96b4-f104b0132650"). InnerVolumeSpecName "kube-api-access-2jbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:14:05.274479 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.274415    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f33c2d6-dd2e-46e0-96b4-f104b0132650" (UID: "8f33c2d6-dd2e-46e0-96b4-f104b0132650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:14:05.373507 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.373471    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c2d6-dd2e-46e0-96b4-f104b0132650-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:14:05.373507 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.373500    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f33c2d6-dd2e-46e0-96b4-f104b0132650-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:14:05.373507 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.373509    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jbgp\" (UniqueName: \"kubernetes.io/projected/8f33c2d6-dd2e-46e0-96b4-f104b0132650-kube-api-access-2jbgp\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:14:05.698192 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.698156    2582 generic.go:356] "Generic (PLEG): container finished" podID="8f33c2d6-dd2e-46e0-96b4-f104b0132650" containerID="ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375" exitCode=0
Jul 11 04:14:05.698340 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.698207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" event={"ID":"8f33c2d6-dd2e-46e0-96b4-f104b0132650","Type":"ContainerDied","Data":"ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375"}
Jul 11 04:14:05.698340 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.698209    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"
Jul 11 04:14:05.698340 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.698250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq" event={"ID":"8f33c2d6-dd2e-46e0-96b4-f104b0132650","Type":"ContainerDied","Data":"ab38b69a0c9c3af88483231b68548ad298447b7e6a74f812214b9144d2711390"}
Jul 11 04:14:05.698340 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.698277    2582 scope.go:117] "RemoveContainer" containerID="ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375"
Jul 11 04:14:05.706902 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.706885    2582 scope.go:117] "RemoveContainer" containerID="ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375"
Jul 11 04:14:05.707231 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:14:05.707210    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375\": container with ID starting with ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375 not found: ID does not exist" containerID="ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375"
Jul 11 04:14:05.707309 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.707237    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375"} err="failed to get container status \"ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375\": rpc error: code = NotFound desc = could not find container \"ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375\": container with ID starting with ad76b55f091276819ce913012847d4628f13ce65cc14e4ffb957502dd17e8375 not found: ID does not exist"
Jul 11 04:14:05.714574 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.714551    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"]
Jul 11 04:14:05.718059 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:05.718039    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7da7c14-6cf55cb6cc-4ddlq"]
Jul 11 04:14:07.607984 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:07.607950    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:14:07.645351 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:07.645319    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f33c2d6-dd2e-46e0-96b4-f104b0132650" path="/var/lib/kubelet/pods/8f33c2d6-dd2e-46e0-96b4-f104b0132650/volumes"
Jul 11 04:14:09.647887 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:09.647855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:15.225362 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:15.225323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:15.326407 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:15.326377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:20.362748 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:20.362713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:20.606336 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:20.606306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:20.734672 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:20.734637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:20.833389 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:20.833348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:20.963031 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:20.963001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:26.581660 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:26.581621    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:26.637516 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:26.637478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:26.719079 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:26.719045    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:26.822722 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:26.822691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:26.955175 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:26.955067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:32.545627 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:32.545583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:32.645493 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:32.645463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:39.505034 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:39.504999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:39.846084 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:39.845980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:40.275889 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:40.275855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:43.101017 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.100966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:43.204572 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.204539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:43.389042 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.388952    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"]
Jul 11 04:14:43.389520 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.389495    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f33c2d6-dd2e-46e0-96b4-f104b0132650" containerName="maas-api"
Jul 11 04:14:43.389520 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.389520    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f33c2d6-dd2e-46e0-96b4-f104b0132650" containerName="maas-api"
Jul 11 04:14:43.389687 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.389633    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f33c2d6-dd2e-46e0-96b4-f104b0132650" containerName="maas-api"
Jul 11 04:14:43.393006 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.392988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.397345 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.397325    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d9f87d6a-serving-cert\""
Jul 11 04:14:43.417732 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.417705    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"]
Jul 11 04:14:43.419994 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.419964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa6c40c-db7f-457b-9747-8cd609fd7151-openshift-service-ca\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.420165 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.420013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdk7r\" (UniqueName: \"kubernetes.io/projected/7aa6c40c-db7f-457b-9747-8cd609fd7151-kube-api-access-vdk7r\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.420165 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.420087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.447938 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.447882    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:43.521219 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.521185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa6c40c-db7f-457b-9747-8cd609fd7151-openshift-service-ca\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.521219 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.521233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdk7r\" (UniqueName: \"kubernetes.io/projected/7aa6c40c-db7f-457b-9747-8cd609fd7151-kube-api-access-vdk7r\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.521486 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.521265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.521486 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:14:43.521418    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-serving-cert: secret "maas-api-e2e-mt-d9f87d6a-serving-cert" not found
Jul 11 04:14:43.521573 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:14:43.521492    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls podName:7aa6c40c-db7f-457b-9747-8cd609fd7151 nodeName:}" failed. No retries permitted until 2026-07-11 04:14:44.021475768 +0000 UTC m=+1928.890922310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls") pod "maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" (UID: "7aa6c40c-db7f-457b-9747-8cd609fd7151") : secret "maas-api-e2e-mt-d9f87d6a-serving-cert" not found
Jul 11 04:14:43.521882 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.521862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa6c40c-db7f-457b-9747-8cd609fd7151-openshift-service-ca\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.531844 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.531813    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdk7r\" (UniqueName: \"kubernetes.io/projected/7aa6c40c-db7f-457b-9747-8cd609fd7151-kube-api-access-vdk7r\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:43.650387 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.650298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:43.876419 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.876387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:43.975058 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:43.974979    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:44.024853 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:44.024822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:44.027358 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:44.027336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls\") pod \"maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:44.303367 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:44.303325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:44.449489 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:44.449466    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"]
Jul 11 04:14:44.453044 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:14:44.452639    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa6c40c_db7f_457b_9747_8cd609fd7151.slice/crio-03a7c678608893023be797cb60538fb943027b4929ca5396c0c397713b0fdd55 WatchSource:0}: Error finding container 03a7c678608893023be797cb60538fb943027b4929ca5396c0c397713b0fdd55: Status 404 returned error can't find the container with id 03a7c678608893023be797cb60538fb943027b4929ca5396c0c397713b0fdd55
Jul 11 04:14:44.896318 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:44.896285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" event={"ID":"7aa6c40c-db7f-457b-9747-8cd609fd7151","Type":"ContainerStarted","Data":"03a7c678608893023be797cb60538fb943027b4929ca5396c0c397713b0fdd55"}
Jul 11 04:14:45.903022 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:45.902980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" event={"ID":"7aa6c40c-db7f-457b-9747-8cd609fd7151","Type":"ContainerStarted","Data":"313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618"}
Jul 11 04:14:45.903556 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:45.903139    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:14:45.920689 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:45.920641    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" podStartSLOduration=2.464259818 podStartE2EDuration="2.920623398s" totalImagesPullingTime="456.36358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:14:45.919559046 +0000 UTC m=+1930.789005613" watchObservedRunningTime="2026-07-11 04:14:45.920623398 +0000 UTC m=+1930.790069961"
Jul 11 04:14:47.660249 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:47.660208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:14:50.002942 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:50.002910    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:14:51.914677 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:14:51.914636    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:15:03.708289 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.708254    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"]
Jul 11 04:15:03.712399 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.712379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.715233 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.715188    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d9f87d6a\"/\"kube-root-ca.crt\""
Jul 11 04:15:03.717176 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.716419    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d9f87d6a\"/\"openshift-service-ca.crt\""
Jul 11 04:15:03.717176 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.716464    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d9f87d6a\"/\"default-dockercfg-fvhr8\""
Jul 11 04:15:03.717176 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.716657    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d9f87d6a\"/\"e2e-lifecycle-model-d9f87d6a-kserve-self-signed-certs\""
Jul 11 04:15:03.719776 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.719738    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"]
Jul 11 04:15:03.739749 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.739714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:03.807053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.807016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-tmp-dir\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.807246 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.807140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50824beb-eaeb-4e04-bb18-58d23604659f-tls-certs\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.807246 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.807194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjqq\" (UniqueName: \"kubernetes.io/projected/50824beb-eaeb-4e04-bb18-58d23604659f-kube-api-access-8mjqq\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.807332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.807255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-dshm\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.807332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.807284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-model-cache\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.807332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.807309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-home\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908026 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.907991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-model-cache\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908026 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-home\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-tmp-dir\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50824beb-eaeb-4e04-bb18-58d23604659f-tls-certs\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjqq\" (UniqueName: \"kubernetes.io/projected/50824beb-eaeb-4e04-bb18-58d23604659f-kube-api-access-8mjqq\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-dshm\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908520 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-model-cache\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908569 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-home\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.908632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.908577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-tmp-dir\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.910408 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.910390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-dshm\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.910670 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.910654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50824beb-eaeb-4e04-bb18-58d23604659f-tls-certs\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:03.917790 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:03.917760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjqq\" (UniqueName: \"kubernetes.io/projected/50824beb-eaeb-4e04-bb18-58d23604659f-kube-api-access-8mjqq\") pod \"e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") " pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:04.024464 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:04.024433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:04.163077 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:04.163054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"]
Jul 11 04:15:04.165159 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:15:04.165131    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50824beb_eaeb_4e04_bb18_58d23604659f.slice/crio-0931d737ed89517ca5bc65787f9db782afc29f9f0a2e97b1380a85a3706065ce WatchSource:0}: Error finding container 0931d737ed89517ca5bc65787f9db782afc29f9f0a2e97b1380a85a3706065ce: Status 404 returned error can't find the container with id 0931d737ed89517ca5bc65787f9db782afc29f9f0a2e97b1380a85a3706065ce
Jul 11 04:15:04.996628 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:04.996591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t" event={"ID":"50824beb-eaeb-4e04-bb18-58d23604659f","Type":"ContainerStarted","Data":"8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e"}
Jul 11 04:15:04.997015 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:04.996641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t" event={"ID":"50824beb-eaeb-4e04-bb18-58d23604659f","Type":"ContainerStarted","Data":"0931d737ed89517ca5bc65787f9db782afc29f9f0a2e97b1380a85a3706065ce"}
Jul 11 04:15:05.014997 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:05.014945    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t" podStartSLOduration=2.014930672 podStartE2EDuration="2.014930672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:15:05.012637732 +0000 UTC m=+1949.882084296" watchObservedRunningTime="2026-07-11 04:15:05.014930672 +0000 UTC m=+1949.884377235"
Jul 11 04:15:07.204406 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:07.204369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:15:14.024874 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:14.024837    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:14.024874 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:14.024884    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:14.037433 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:14.037407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:14.052812 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:14.052775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:15:18.065029 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:18.064981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:29.618684 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:29.618593    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:29.720660 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:29.720631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:30.204503 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:30.204468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:15:36.803755 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:36.803717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:36.903728 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:36.903697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:46.701758 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:46.701710    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"]
Jul 11 04:15:46.702299 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:46.701965    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" podUID="7aa6c40c-db7f-457b-9747-8cd609fd7151" containerName="maas-api" containerID="cri-o://313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618" gracePeriod=30
Jul 11 04:15:46.729072 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:46.729047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:46.967437 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:46.967380    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:15:47.116562 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.116525    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls\") pod \"7aa6c40c-db7f-457b-9747-8cd609fd7151\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") "
Jul 11 04:15:47.116734 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.116603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa6c40c-db7f-457b-9747-8cd609fd7151-openshift-service-ca\") pod \"7aa6c40c-db7f-457b-9747-8cd609fd7151\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") "
Jul 11 04:15:47.116734 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.116641    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdk7r\" (UniqueName: \"kubernetes.io/projected/7aa6c40c-db7f-457b-9747-8cd609fd7151-kube-api-access-vdk7r\") pod \"7aa6c40c-db7f-457b-9747-8cd609fd7151\" (UID: \"7aa6c40c-db7f-457b-9747-8cd609fd7151\") "
Jul 11 04:15:47.116951 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.116927    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa6c40c-db7f-457b-9747-8cd609fd7151-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7aa6c40c-db7f-457b-9747-8cd609fd7151" (UID: "7aa6c40c-db7f-457b-9747-8cd609fd7151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:15:47.118922 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.118888    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7aa6c40c-db7f-457b-9747-8cd609fd7151" (UID: "7aa6c40c-db7f-457b-9747-8cd609fd7151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:15:47.118922 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.118899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa6c40c-db7f-457b-9747-8cd609fd7151-kube-api-access-vdk7r" (OuterVolumeSpecName: "kube-api-access-vdk7r") pod "7aa6c40c-db7f-457b-9747-8cd609fd7151" (UID: "7aa6c40c-db7f-457b-9747-8cd609fd7151"). InnerVolumeSpecName "kube-api-access-vdk7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:15:47.217901 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.217817    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aa6c40c-db7f-457b-9747-8cd609fd7151-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:15:47.217901 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.217841    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa6c40c-db7f-457b-9747-8cd609fd7151-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:15:47.217901 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.217851    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdk7r\" (UniqueName: \"kubernetes.io/projected/7aa6c40c-db7f-457b-9747-8cd609fd7151-kube-api-access-vdk7r\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:15:47.220738 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.220708    2582 generic.go:356] "Generic (PLEG): container finished" podID="7aa6c40c-db7f-457b-9747-8cd609fd7151" containerID="313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618" exitCode=0
Jul 11 04:15:47.220853 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.220758    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"
Jul 11 04:15:47.220853 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.220797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" event={"ID":"7aa6c40c-db7f-457b-9747-8cd609fd7151","Type":"ContainerDied","Data":"313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618"}
Jul 11 04:15:47.220924 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.220855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" event={"ID":"7aa6c40c-db7f-457b-9747-8cd609fd7151","Type":"ContainerDied","Data":"03a7c678608893023be797cb60538fb943027b4929ca5396c0c397713b0fdd55"}
Jul 11 04:15:47.220924 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.220877    2582 scope.go:117] "RemoveContainer" containerID="313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618"
Jul 11 04:15:47.229930 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.229912    2582 scope.go:117] "RemoveContainer" containerID="313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618"
Jul 11 04:15:47.230218 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:15:47.230197    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618\": container with ID starting with 313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618 not found: ID does not exist" containerID="313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618"
Jul 11 04:15:47.230286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.230227    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618"} err="failed to get container status \"313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618\": rpc error: code = NotFound desc = could not find container \"313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618\": container with ID starting with 313c8ba96e1c1c0d27bad4281a410d8be02eedb84ee9f30ef876690671bf8618 not found: ID does not exist"
Jul 11 04:15:47.241433 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.241405    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"]
Jul 11 04:15:47.246434 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.246417    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82"]
Jul 11 04:15:47.654551 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:47.654517    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa6c40c-db7f-457b-9747-8cd609fd7151" path="/var/lib/kubelet/pods/7aa6c40c-db7f-457b-9747-8cd609fd7151/volumes"
Jul 11 04:15:49.908561 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:49.908453    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9f87d6a-6fdb948ff9-j8g82" podUID="7aa6c40c-db7f-457b-9747-8cd609fd7151" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.54:8443/health\": context deadline exceeded"
Jul 11 04:15:50.205692 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:50.205603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:15:51.391563 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:51.391526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:15:56.376838 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:56.376802    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"]
Jul 11 04:15:56.420954 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:15:56.420919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:11.398030 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.397958    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t" podUID="50824beb-eaeb-4e04-bb18-58d23604659f" containerName="main" containerID="cri-o://8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e" gracePeriod=45
Jul 11 04:16:11.668140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.668091    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:16:11.832436 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832399    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-dshm\") pod \"50824beb-eaeb-4e04-bb18-58d23604659f\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") "
Jul 11 04:16:11.832436 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-home\") pod \"50824beb-eaeb-4e04-bb18-58d23604659f\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") "
Jul 11 04:16:11.832694 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832469    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-model-cache\") pod \"50824beb-eaeb-4e04-bb18-58d23604659f\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") "
Jul 11 04:16:11.832694 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832510    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-tmp-dir\") pod \"50824beb-eaeb-4e04-bb18-58d23604659f\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") "
Jul 11 04:16:11.832694 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50824beb-eaeb-4e04-bb18-58d23604659f-tls-certs\") pod \"50824beb-eaeb-4e04-bb18-58d23604659f\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") "
Jul 11 04:16:11.832694 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mjqq\" (UniqueName: \"kubernetes.io/projected/50824beb-eaeb-4e04-bb18-58d23604659f-kube-api-access-8mjqq\") pod \"50824beb-eaeb-4e04-bb18-58d23604659f\" (UID: \"50824beb-eaeb-4e04-bb18-58d23604659f\") "
Jul 11 04:16:11.832893 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-home" (OuterVolumeSpecName: "home") pod "50824beb-eaeb-4e04-bb18-58d23604659f" (UID: "50824beb-eaeb-4e04-bb18-58d23604659f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:16:11.832893 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832850    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:16:11.832893 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-model-cache" (OuterVolumeSpecName: "model-cache") pod "50824beb-eaeb-4e04-bb18-58d23604659f" (UID: "50824beb-eaeb-4e04-bb18-58d23604659f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:16:11.833015 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.832890    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50824beb-eaeb-4e04-bb18-58d23604659f" (UID: "50824beb-eaeb-4e04-bb18-58d23604659f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:16:11.834905 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.834882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-dshm" (OuterVolumeSpecName: "dshm") pod "50824beb-eaeb-4e04-bb18-58d23604659f" (UID: "50824beb-eaeb-4e04-bb18-58d23604659f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:16:11.835006 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.834939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50824beb-eaeb-4e04-bb18-58d23604659f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50824beb-eaeb-4e04-bb18-58d23604659f" (UID: "50824beb-eaeb-4e04-bb18-58d23604659f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:16:11.835006 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.834952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50824beb-eaeb-4e04-bb18-58d23604659f-kube-api-access-8mjqq" (OuterVolumeSpecName: "kube-api-access-8mjqq") pod "50824beb-eaeb-4e04-bb18-58d23604659f" (UID: "50824beb-eaeb-4e04-bb18-58d23604659f"). InnerVolumeSpecName "kube-api-access-8mjqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:16:11.933848 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.933761    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjqq\" (UniqueName: \"kubernetes.io/projected/50824beb-eaeb-4e04-bb18-58d23604659f-kube-api-access-8mjqq\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:16:11.933848 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.933792    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:16:11.933848 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.933804    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:16:11.933848 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.933814    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50824beb-eaeb-4e04-bb18-58d23604659f-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:16:11.933848 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:11.933822    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50824beb-eaeb-4e04-bb18-58d23604659f-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:16:12.349065 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.349028    2582 generic.go:356] "Generic (PLEG): container finished" podID="50824beb-eaeb-4e04-bb18-58d23604659f" containerID="8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e" exitCode=0
Jul 11 04:16:12.349065 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.349069    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"
Jul 11 04:16:12.349332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.349127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t" event={"ID":"50824beb-eaeb-4e04-bb18-58d23604659f","Type":"ContainerDied","Data":"8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e"}
Jul 11 04:16:12.349332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.349168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t" event={"ID":"50824beb-eaeb-4e04-bb18-58d23604659f","Type":"ContainerDied","Data":"0931d737ed89517ca5bc65787f9db782afc29f9f0a2e97b1380a85a3706065ce"}
Jul 11 04:16:12.349332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.349189    2582 scope.go:117] "RemoveContainer" containerID="8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e"
Jul 11 04:16:12.358350 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.358332    2582 scope.go:117] "RemoveContainer" containerID="8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e"
Jul 11 04:16:12.358621 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:16:12.358604    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e\": container with ID starting with 8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e not found: ID does not exist" containerID="8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e"
Jul 11 04:16:12.358678 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.358629    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e"} err="failed to get container status \"8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e\": rpc error: code = NotFound desc = could not find container \"8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e\": container with ID starting with 8459383ec989d958f2a29ab3af148fcf4df949cf5769b700791dae996d2a770e not found: ID does not exist"
Jul 11 04:16:12.371365 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.371331    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"]
Jul 11 04:16:12.375251 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:12.375222    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d9f87d6a/e2e-lifecycle-model-d9f87d6a-kserve-5466b7f97f-xb45t"]
Jul 11 04:16:13.646721 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:13.646693    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50824beb-eaeb-4e04-bb18-58d23604659f" path="/var/lib/kubelet/pods/50824beb-eaeb-4e04-bb18-58d23604659f/volumes"
Jul 11 04:16:33.081008 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:33.080976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:33.468651 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:33.468567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:39.206301 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:39.206256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:44.655800 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:44.655763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:44.713922 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:44.713891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:52.074860 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:52.074817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:53.687599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:53.687570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:53.924565 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:53.924530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:54.969934 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:54.969906    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:55.072073 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:55.072032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:55.206985 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:55.206955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:55.903888 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:55.903852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:56.003834 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:56.003804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:56.103767 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:56.103736    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:56.827189 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:56.827089    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:56.927137 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:56.927080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:57.494547 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:57.494501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:16:57.817484 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:16:57.817394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:17:03.216311 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:03.216278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:03.315176 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:03.315144    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:03.908441 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:03.908411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:09.669958 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:09.669912    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:09.773508 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:09.773457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:09.871345 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:09.871319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:11.607255 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:11.607225    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:17:20.365908 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:20.365847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:20.713035 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:20.712950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:35.865282 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:35.865248    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:17:35.873373 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:35.873352    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:17:41.673964 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:41.673930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:47.217993 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:47.217958    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:47.319649 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:47.319609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:47.569412 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:47.569375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:47.918310 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:47.918228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:48.015477 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:48.015443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.001229 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.001186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.078974 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.078941    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"]
Jul 11 04:17:51.079575 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.079533    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50824beb-eaeb-4e04-bb18-58d23604659f" containerName="main"
Jul 11 04:17:51.079575 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.079560    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50824beb-eaeb-4e04-bb18-58d23604659f" containerName="main"
Jul 11 04:17:51.079575 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.079579    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa6c40c-db7f-457b-9747-8cd609fd7151" containerName="maas-api"
Jul 11 04:17:51.079830 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.079588    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa6c40c-db7f-457b-9747-8cd609fd7151" containerName="maas-api"
Jul 11 04:17:51.079830 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.079704    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50824beb-eaeb-4e04-bb18-58d23604659f" containerName="main"
Jul 11 04:17:51.079830 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.079718    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa6c40c-db7f-457b-9747-8cd609fd7151" containerName="maas-api"
Jul 11 04:17:51.083320 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.083299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.084555 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.084518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.085630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.085612    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-888089-serving-cert\""
Jul 11 04:17:51.092979 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.092953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"]
Jul 11 04:17:51.123728 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.123685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.123920 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.123739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e8375-6159-44b8-9af1-b01a841c14cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.123920 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.123774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjbl\" (UniqueName: \"kubernetes.io/projected/7d7e8375-6159-44b8-9af1-b01a841c14cb-kube-api-access-9mjbl\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.142924 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.142894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.224560 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.224525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.224747 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.224567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e8375-6159-44b8-9af1-b01a841c14cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.224747 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:17:51.224668    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-serving-cert: secret "maas-api-e2e-inf-a-888089-serving-cert" not found
Jul 11 04:17:51.224747 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:17:51.224738    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls podName:7d7e8375-6159-44b8-9af1-b01a841c14cb nodeName:}" failed. No retries permitted until 2026-07-11 04:17:51.724720286 +0000 UTC m=+2116.594166828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls") pod "maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" (UID: "7d7e8375-6159-44b8-9af1-b01a841c14cb") : secret "maas-api-e2e-inf-a-888089-serving-cert" not found
Jul 11 04:17:51.224911 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.224827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjbl\" (UniqueName: \"kubernetes.io/projected/7d7e8375-6159-44b8-9af1-b01a841c14cb-kube-api-access-9mjbl\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.225304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.225284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e8375-6159-44b8-9af1-b01a841c14cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.240718 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.240683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjbl\" (UniqueName: \"kubernetes.io/projected/7d7e8375-6159-44b8-9af1-b01a841c14cb-kube-api-access-9mjbl\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.562908 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.562874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.665134 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.665086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.727954 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.727922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.730436 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.730416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:51.765790 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.765761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:51.995004 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:51.994971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:52.145415 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:52.145377    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"]
Jul 11 04:17:52.147331 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:17:52.147294    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7e8375_6159_44b8_9af1_b01a841c14cb.slice/crio-27efb1affbce645e5371bb5322ee7504b51c96b31ae7fdf6d54aea5184b3f62a WatchSource:0}: Error finding container 27efb1affbce645e5371bb5322ee7504b51c96b31ae7fdf6d54aea5184b3f62a: Status 404 returned error can't find the container with id 27efb1affbce645e5371bb5322ee7504b51c96b31ae7fdf6d54aea5184b3f62a
Jul 11 04:17:52.842520 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:52.842428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" event={"ID":"7d7e8375-6159-44b8-9af1-b01a841c14cb","Type":"ContainerStarted","Data":"8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33"}
Jul 11 04:17:52.842520 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:52.842475    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" event={"ID":"7d7e8375-6159-44b8-9af1-b01a841c14cb","Type":"ContainerStarted","Data":"27efb1affbce645e5371bb5322ee7504b51c96b31ae7fdf6d54aea5184b3f62a"}
Jul 11 04:17:52.842735 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:52.842569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:52.859538 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:52.859485    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" podStartSLOduration=1.423486863 podStartE2EDuration="1.8594673s" totalImagesPullingTime="435.980437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:17:52.857117216 +0000 UTC m=+2117.726563780" watchObservedRunningTime="2026-07-11 04:17:52.8594673 +0000 UTC m=+2117.728913863"
Jul 11 04:17:55.540228 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:55.540196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:55.880620 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:55.880525    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:56.201592 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:56.201501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:56.695484 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:56.695448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:58.853529 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:58.853497    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:17:59.022478 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.022442    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:17:59.033517 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.033483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:59.099310 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.099277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:59.103762 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.103686    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"]
Jul 11 04:17:59.107407 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.107385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.109915 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.109896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-075325-serving-cert\""
Jul 11 04:17:59.117135 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.117085    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"]
Jul 11 04:17:59.170048 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.170012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:59.194543 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.194508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hj52\" (UniqueName: \"kubernetes.io/projected/97c27d51-f696-4d2c-a456-3478bde9112f-kube-api-access-8hj52\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.194712 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.194555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.194712 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.194636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c27d51-f696-4d2c-a456-3478bde9112f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.295192 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.295154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c27d51-f696-4d2c-a456-3478bde9112f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.295377 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.295265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hj52\" (UniqueName: \"kubernetes.io/projected/97c27d51-f696-4d2c-a456-3478bde9112f-kube-api-access-8hj52\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.295377 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.295306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.295525 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:17:59.295444    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-serving-cert: secret "maas-api-e2e-inf-b-075325-serving-cert" not found
Jul 11 04:17:59.295525 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:17:59.295516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls podName:97c27d51-f696-4d2c-a456-3478bde9112f nodeName:}" failed. No retries permitted until 2026-07-11 04:17:59.795495874 +0000 UTC m=+2124.664942420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls") pod "maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" (UID: "97c27d51-f696-4d2c-a456-3478bde9112f") : secret "maas-api-e2e-inf-b-075325-serving-cert" not found
Jul 11 04:17:59.295899 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.295877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c27d51-f696-4d2c-a456-3478bde9112f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.310224 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.310196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hj52\" (UniqueName: \"kubernetes.io/projected/97c27d51-f696-4d2c-a456-3478bde9112f-kube-api-access-8hj52\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.671747 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.671712    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:59.774275 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.774244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:17:59.800651 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:17:59.800619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:17:59.800775 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:17:59.800765    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-serving-cert: secret "maas-api-e2e-inf-b-075325-serving-cert" not found
Jul 11 04:17:59.800836 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:17:59.800822    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls podName:97c27d51-f696-4d2c-a456-3478bde9112f nodeName:}" failed. No retries permitted until 2026-07-11 04:18:00.800805657 +0000 UTC m=+2125.670252199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls") pod "maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" (UID: "97c27d51-f696-4d2c-a456-3478bde9112f") : secret "maas-api-e2e-inf-b-075325-serving-cert" not found
Jul 11 04:18:00.539051 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:00.539009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:00.809904 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:00.809816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:18:00.812376 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:00.812352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") pod \"maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:18:00.919599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:00.919563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:18:01.062119 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:01.062069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"]
Jul 11 04:18:01.063613 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:18:01.063588    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c27d51_f696_4d2c_a456_3478bde9112f.slice/crio-6bef2777b2d789663c850723ce010a350345b0bf22f767eb967f142a750a6059 WatchSource:0}: Error finding container 6bef2777b2d789663c850723ce010a350345b0bf22f767eb967f142a750a6059: Status 404 returned error can't find the container with id 6bef2777b2d789663c850723ce010a350345b0bf22f767eb967f142a750a6059
Jul 11 04:18:01.896630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:01.896541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" event={"ID":"97c27d51-f696-4d2c-a456-3478bde9112f","Type":"ContainerStarted","Data":"79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940"}
Jul 11 04:18:01.896630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:01.896598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" event={"ID":"97c27d51-f696-4d2c-a456-3478bde9112f","Type":"ContainerStarted","Data":"6bef2777b2d789663c850723ce010a350345b0bf22f767eb967f142a750a6059"}
Jul 11 04:18:01.896630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:01.896631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:18:01.913657 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:01.913608    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" podStartSLOduration=2.457055542 podStartE2EDuration="2.913591071s" totalImagesPullingTime="456.535529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:18:01.912507937 +0000 UTC m=+2126.781954502" watchObservedRunningTime="2026-07-11 04:18:01.913591071 +0000 UTC m=+2126.783037687"
Jul 11 04:18:03.566905 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:03.566871    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:07.907501 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:07.907471    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:18:10.308666 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:10.308617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:18:12.357671 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.357628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:12.458296 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.458260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:12.609189 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.609088    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"]
Jul 11 04:18:12.612459 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.612432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.615716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.615678    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-888089\"/\"openshift-service-ca.crt\""
Jul 11 04:18:12.615716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.615695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-888089\"/\"kube-root-ca.crt\""
Jul 11 04:18:12.615716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.615687    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-888089\"/\"default-dockercfg-pgngn\""
Jul 11 04:18:12.615992 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.615774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-888089\"/\"test-model-888089-kserve-self-signed-certs\""
Jul 11 04:18:12.620339 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.620313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"]
Jul 11 04:18:12.638895 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.638866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:12.722536 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.722499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9cg\" (UniqueName: \"kubernetes.io/projected/c0de6226-b41e-41fb-8116-607ba3375e28-kube-api-access-hc9cg\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.722726 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.722543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-tmp-dir\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.722726 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.722576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-dshm\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.722726 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.722665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-home\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.722726 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.722704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0de6226-b41e-41fb-8116-607ba3375e28-tls-certs\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.722898 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.722734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-model-cache\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.823856 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.823819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-home\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824062 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.823864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0de6226-b41e-41fb-8116-607ba3375e28-tls-certs\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824182 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-model-cache\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824247 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9cg\" (UniqueName: \"kubernetes.io/projected/c0de6226-b41e-41fb-8116-607ba3375e28-kube-api-access-hc9cg\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824247 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-tmp-dir\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824352 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-dshm\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824412 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-home\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824412 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-model-cache\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.824585 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.824560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-tmp-dir\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.826493 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.826472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-dshm\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.826663 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.826642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0de6226-b41e-41fb-8116-607ba3375e28-tls-certs\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.831199 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.831179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9cg\" (UniqueName: \"kubernetes.io/projected/c0de6226-b41e-41fb-8116-607ba3375e28-kube-api-access-hc9cg\") pod \"test-model-888089-kserve-55cb65985f-h7lj2\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") " pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:12.925250 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:12.925169    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:13.067941 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:13.067919    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"]
Jul 11 04:18:13.069952 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:18:13.069921    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0de6226_b41e_41fb_8116_607ba3375e28.slice/crio-14ff81b1de94985e346a9ee8b10c6b792729811b386a30147af4a597be55bb5c WatchSource:0}: Error finding container 14ff81b1de94985e346a9ee8b10c6b792729811b386a30147af4a597be55bb5c: Status 404 returned error can't find the container with id 14ff81b1de94985e346a9ee8b10c6b792729811b386a30147af4a597be55bb5c
Jul 11 04:18:13.961129 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:13.961076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2" event={"ID":"c0de6226-b41e-41fb-8116-607ba3375e28","Type":"ContainerStarted","Data":"8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65"}
Jul 11 04:18:13.961618 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:13.961139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2" event={"ID":"c0de6226-b41e-41fb-8116-607ba3375e28","Type":"ContainerStarted","Data":"14ff81b1de94985e346a9ee8b10c6b792729811b386a30147af4a597be55bb5c"}
Jul 11 04:18:13.977224 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:13.977179    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2" podStartSLOduration=1.9771637800000001 podStartE2EDuration="1.97716378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:18:13.976242452 +0000 UTC m=+2138.845689040" watchObservedRunningTime="2026-07-11 04:18:13.97716378 +0000 UTC m=+2138.846610345"
Jul 11 04:18:18.008011 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:18.007981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:18.713309 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:18.713273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:18:20.383344 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:20.383309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:20.485646 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:20.485614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:22.925604 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:22.925569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:22.926001 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:22.925642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:22.938363 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:22.938337    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:23.015137 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:23.015083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:18:53.193479 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.193445    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"]
Jul 11 04:18:53.197018 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.196997    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.200298 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.200273    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-075325\"/\"test-model-075325-kserve-self-signed-certs\""
Jul 11 04:18:53.200298 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.200282    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-075325\"/\"kube-root-ca.crt\""
Jul 11 04:18:53.200499 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.200356    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-075325\"/\"openshift-service-ca.crt\""
Jul 11 04:18:53.200562 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.200508    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-075325\"/\"default-dockercfg-289s8\""
Jul 11 04:18:53.208739 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.208716    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"]
Jul 11 04:18:53.226922 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.226885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:18:53.309215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.309176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-dshm\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.309215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.309218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-model-cache\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.309457 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.309319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88838224-c570-45f7-89ef-034619564318-tls-certs\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.309457 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.309350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-home\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.309457 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.309371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7twb\" (UniqueName: \"kubernetes.io/projected/88838224-c570-45f7-89ef-034619564318-kube-api-access-d7twb\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.309457 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.309406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-tmp-dir\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.410715 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.410669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88838224-c570-45f7-89ef-034619564318-tls-certs\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.410715 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.410716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-home\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.410976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.410754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7twb\" (UniqueName: \"kubernetes.io/projected/88838224-c570-45f7-89ef-034619564318-kube-api-access-d7twb\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.410976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.410795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-tmp-dir\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.410976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.410859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-dshm\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.410976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.410891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-model-cache\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.411329 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.411296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-home\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.411329 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.411317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-tmp-dir\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.411329 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.411325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-model-cache\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.413286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.413257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-dshm\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.413482 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.413466    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88838224-c570-45f7-89ef-034619564318-tls-certs\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.419917 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.419895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7twb\" (UniqueName: \"kubernetes.io/projected/88838224-c570-45f7-89ef-034619564318-kube-api-access-d7twb\") pod \"test-model-075325-kserve-7d9cc9cc89-d46nt\" (UID: \"88838224-c570-45f7-89ef-034619564318\") " pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.507860 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.507822    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:18:53.655438 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:53.655401    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"]
Jul 11 04:18:53.657390 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:18:53.657356    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88838224_c570_45f7_89ef_034619564318.slice/crio-132a0634585f8cd87b180f579cde59ef80c1d53ad02131fa93645e1aa0a43224 WatchSource:0}: Error finding container 132a0634585f8cd87b180f579cde59ef80c1d53ad02131fa93645e1aa0a43224: Status 404 returned error can't find the container with id 132a0634585f8cd87b180f579cde59ef80c1d53ad02131fa93645e1aa0a43224
Jul 11 04:18:54.169017 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:54.168981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt" event={"ID":"88838224-c570-45f7-89ef-034619564318","Type":"ContainerStarted","Data":"39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae"}
Jul 11 04:18:54.169215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:54.169026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt" event={"ID":"88838224-c570-45f7-89ef-034619564318","Type":"ContainerStarted","Data":"132a0634585f8cd87b180f579cde59ef80c1d53ad02131fa93645e1aa0a43224"}
Jul 11 04:18:54.185233 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:54.185190    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt" podStartSLOduration=1.185174441 podStartE2EDuration="1.185174441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:18:54.18315405 +0000 UTC m=+2179.052600614" watchObservedRunningTime="2026-07-11 04:18:54.185174441 +0000 UTC m=+2179.054621004"
Jul 11 04:18:57.211574 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:18:57.211531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:19:03.508846 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:03.508809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:19:03.508846 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:03.508853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:19:03.521863 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:03.521841    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:19:04.232263 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:04.232236    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:19:12.400091 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:12.400051    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"]
Jul 11 04:19:12.415929 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:12.415892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:13.335952 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:13.335917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"]
Jul 11 04:19:13.353416 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:13.353380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:13.944215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:13.944183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:14.048223 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:14.048185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:15.806332 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:15.806300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:19:23.840303 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:23.840269    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"]
Jul 11 04:19:23.840906 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:23.840483    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" podUID="7d7e8375-6159-44b8-9af1-b01a841c14cb" containerName="maas-api" containerID="cri-o://8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33" gracePeriod=30
Jul 11 04:19:23.848253 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:23.848222    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" podUID="7d7e8375-6159-44b8-9af1-b01a841c14cb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.56:8443/health\": dial tcp 10.132.0.56:8443: connect: connection refused"
Jul 11 04:19:23.883741 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:23.883707    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:24.291545 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.291514    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:19:24.326084 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.326046    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d7e8375-6159-44b8-9af1-b01a841c14cb" containerID="8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33" exitCode=0
Jul 11 04:19:24.326255 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.326086    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"
Jul 11 04:19:24.326255 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.326120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" event={"ID":"7d7e8375-6159-44b8-9af1-b01a841c14cb","Type":"ContainerDied","Data":"8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33"}
Jul 11 04:19:24.326255 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.326181    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq" event={"ID":"7d7e8375-6159-44b8-9af1-b01a841c14cb","Type":"ContainerDied","Data":"27efb1affbce645e5371bb5322ee7504b51c96b31ae7fdf6d54aea5184b3f62a"}
Jul 11 04:19:24.326255 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.326214    2582 scope.go:117] "RemoveContainer" containerID="8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33"
Jul 11 04:19:24.335154 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.335083    2582 scope.go:117] "RemoveContainer" containerID="8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33"
Jul 11 04:19:24.335462 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:19:24.335443    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33\": container with ID starting with 8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33 not found: ID does not exist" containerID="8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33"
Jul 11 04:19:24.335535 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.335471    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33"} err="failed to get container status \"8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33\": rpc error: code = NotFound desc = could not find container \"8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33\": container with ID starting with 8bcfd3804d57dfb43c1bcf561edcee4d14cf5f8f4c37c678f7e3b18e13b0df33 not found: ID does not exist"
Jul 11 04:19:24.399444 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.399348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls\") pod \"7d7e8375-6159-44b8-9af1-b01a841c14cb\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") "
Jul 11 04:19:24.399444 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.399384    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e8375-6159-44b8-9af1-b01a841c14cb-openshift-service-ca\") pod \"7d7e8375-6159-44b8-9af1-b01a841c14cb\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") "
Jul 11 04:19:24.399444 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.399407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mjbl\" (UniqueName: \"kubernetes.io/projected/7d7e8375-6159-44b8-9af1-b01a841c14cb-kube-api-access-9mjbl\") pod \"7d7e8375-6159-44b8-9af1-b01a841c14cb\" (UID: \"7d7e8375-6159-44b8-9af1-b01a841c14cb\") "
Jul 11 04:19:24.399720 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.399698    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7e8375-6159-44b8-9af1-b01a841c14cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d7e8375-6159-44b8-9af1-b01a841c14cb" (UID: "7d7e8375-6159-44b8-9af1-b01a841c14cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:19:24.401740 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.401713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7e8375-6159-44b8-9af1-b01a841c14cb-kube-api-access-9mjbl" (OuterVolumeSpecName: "kube-api-access-9mjbl") pod "7d7e8375-6159-44b8-9af1-b01a841c14cb" (UID: "7d7e8375-6159-44b8-9af1-b01a841c14cb"). InnerVolumeSpecName "kube-api-access-9mjbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:19:24.401855 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.401756    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d7e8375-6159-44b8-9af1-b01a841c14cb" (UID: "7d7e8375-6159-44b8-9af1-b01a841c14cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:19:24.500039 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.500005    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7e8375-6159-44b8-9af1-b01a841c14cb-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:24.500039 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.500033    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7e8375-6159-44b8-9af1-b01a841c14cb-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:24.500039 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.500043    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjbl\" (UniqueName: \"kubernetes.io/projected/7d7e8375-6159-44b8-9af1-b01a841c14cb-kube-api-access-9mjbl\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:24.647515 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.647487    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"]
Jul 11 04:19:24.652417 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:24.652350    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-888089-5fdb7d86c9-7ngpq"]
Jul 11 04:19:25.646444 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:25.646414    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d7e8375-6159-44b8-9af1-b01a841c14cb" path="/var/lib/kubelet/pods/7d7e8375-6159-44b8-9af1-b01a841c14cb/volumes"
Jul 11 04:19:27.423236 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.423192    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2" podUID="c0de6226-b41e-41fb-8116-607ba3375e28" containerName="main" containerID="cri-o://8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65" gracePeriod=45
Jul 11 04:19:27.690710 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.690680    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:19:27.728970 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.728940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-home\") pod \"c0de6226-b41e-41fb-8116-607ba3375e28\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") "
Jul 11 04:19:27.728970 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.728971    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0de6226-b41e-41fb-8116-607ba3375e28-tls-certs\") pod \"c0de6226-b41e-41fb-8116-607ba3375e28\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") "
Jul 11 04:19:27.729241 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-tmp-dir\") pod \"c0de6226-b41e-41fb-8116-607ba3375e28\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") "
Jul 11 04:19:27.729241 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729054    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-dshm\") pod \"c0de6226-b41e-41fb-8116-607ba3375e28\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") "
Jul 11 04:19:27.729241 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729093    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-model-cache\") pod \"c0de6226-b41e-41fb-8116-607ba3375e28\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") "
Jul 11 04:19:27.729241 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc9cg\" (UniqueName: \"kubernetes.io/projected/c0de6226-b41e-41fb-8116-607ba3375e28-kube-api-access-hc9cg\") pod \"c0de6226-b41e-41fb-8116-607ba3375e28\" (UID: \"c0de6226-b41e-41fb-8116-607ba3375e28\") "
Jul 11 04:19:27.729440 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729296    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-home" (OuterVolumeSpecName: "home") pod "c0de6226-b41e-41fb-8116-607ba3375e28" (UID: "c0de6226-b41e-41fb-8116-607ba3375e28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:27.729535 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729479    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0de6226-b41e-41fb-8116-607ba3375e28" (UID: "c0de6226-b41e-41fb-8116-607ba3375e28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:27.729611 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729565    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0de6226-b41e-41fb-8116-607ba3375e28" (UID: "c0de6226-b41e-41fb-8116-607ba3375e28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:27.729611 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729585    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:27.729611 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.729603    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:27.731503 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.731473    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0de6226-b41e-41fb-8116-607ba3375e28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0de6226-b41e-41fb-8116-607ba3375e28" (UID: "c0de6226-b41e-41fb-8116-607ba3375e28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:19:27.731503 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.731480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-dshm" (OuterVolumeSpecName: "dshm") pod "c0de6226-b41e-41fb-8116-607ba3375e28" (UID: "c0de6226-b41e-41fb-8116-607ba3375e28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:27.731659 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.731507    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0de6226-b41e-41fb-8116-607ba3375e28-kube-api-access-hc9cg" (OuterVolumeSpecName: "kube-api-access-hc9cg") pod "c0de6226-b41e-41fb-8116-607ba3375e28" (UID: "c0de6226-b41e-41fb-8116-607ba3375e28"). InnerVolumeSpecName "kube-api-access-hc9cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:19:27.830682 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.830652    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:27.830682 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.830679    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc9cg\" (UniqueName: \"kubernetes.io/projected/c0de6226-b41e-41fb-8116-607ba3375e28-kube-api-access-hc9cg\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:27.830682 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.830690    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0de6226-b41e-41fb-8116-607ba3375e28-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:27.830920 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:27.830699    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0de6226-b41e-41fb-8116-607ba3375e28-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:28.349140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.349076    2582 generic.go:356] "Generic (PLEG): container finished" podID="c0de6226-b41e-41fb-8116-607ba3375e28" containerID="8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65" exitCode=0
Jul 11 04:19:28.349140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.349138    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"
Jul 11 04:19:28.349346 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.349134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2" event={"ID":"c0de6226-b41e-41fb-8116-607ba3375e28","Type":"ContainerDied","Data":"8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65"}
Jul 11 04:19:28.349346 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.349247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2" event={"ID":"c0de6226-b41e-41fb-8116-607ba3375e28","Type":"ContainerDied","Data":"14ff81b1de94985e346a9ee8b10c6b792729811b386a30147af4a597be55bb5c"}
Jul 11 04:19:28.349346 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.349266    2582 scope.go:117] "RemoveContainer" containerID="8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65"
Jul 11 04:19:28.358844 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.358827    2582 scope.go:117] "RemoveContainer" containerID="8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65"
Jul 11 04:19:28.359086 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:19:28.359069    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65\": container with ID starting with 8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65 not found: ID does not exist" containerID="8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65"
Jul 11 04:19:28.359214 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.359109    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65"} err="failed to get container status \"8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65\": rpc error: code = NotFound desc = could not find container \"8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65\": container with ID starting with 8e4d1e1a4f2d0ee8080ea2ec471a9b210fa52dc7af5cbe8a6716d8a707085c65 not found: ID does not exist"
Jul 11 04:19:28.360290 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.360248    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt" podUID="88838224-c570-45f7-89ef-034619564318" containerName="main" containerID="cri-o://39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae" gracePeriod=45
Jul 11 04:19:28.370475 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.370453    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"]
Jul 11 04:19:28.373346 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.373324    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-888089/test-model-888089-kserve-55cb65985f-h7lj2"]
Jul 11 04:19:28.550015 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.549982    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:28.620938 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.620913    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:19:28.637009 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.636978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88838224-c570-45f7-89ef-034619564318-tls-certs\") pod \"88838224-c570-45f7-89ef-034619564318\" (UID: \"88838224-c570-45f7-89ef-034619564318\") "
Jul 11 04:19:28.637202 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-tmp-dir\") pod \"88838224-c570-45f7-89ef-034619564318\" (UID: \"88838224-c570-45f7-89ef-034619564318\") "
Jul 11 04:19:28.637202 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-dshm\") pod \"88838224-c570-45f7-89ef-034619564318\" (UID: \"88838224-c570-45f7-89ef-034619564318\") "
Jul 11 04:19:28.637202 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-model-cache\") pod \"88838224-c570-45f7-89ef-034619564318\" (UID: \"88838224-c570-45f7-89ef-034619564318\") "
Jul 11 04:19:28.637202 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7twb\" (UniqueName: \"kubernetes.io/projected/88838224-c570-45f7-89ef-034619564318-kube-api-access-d7twb\") pod \"88838224-c570-45f7-89ef-034619564318\" (UID: \"88838224-c570-45f7-89ef-034619564318\") "
Jul 11 04:19:28.637428 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-home\") pod \"88838224-c570-45f7-89ef-034619564318\" (UID: \"88838224-c570-45f7-89ef-034619564318\") "
Jul 11 04:19:28.637869 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637453    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88838224-c570-45f7-89ef-034619564318" (UID: "88838224-c570-45f7-89ef-034619564318"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:28.637869 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637776    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-home" (OuterVolumeSpecName: "home") pod "88838224-c570-45f7-89ef-034619564318" (UID: "88838224-c570-45f7-89ef-034619564318"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:28.637869 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637812    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:28.637869 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.637839    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-model-cache" (OuterVolumeSpecName: "model-cache") pod "88838224-c570-45f7-89ef-034619564318" (UID: "88838224-c570-45f7-89ef-034619564318"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:28.640177 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.640153    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88838224-c570-45f7-89ef-034619564318-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88838224-c570-45f7-89ef-034619564318" (UID: "88838224-c570-45f7-89ef-034619564318"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:19:28.640433 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.640412    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88838224-c570-45f7-89ef-034619564318-kube-api-access-d7twb" (OuterVolumeSpecName: "kube-api-access-d7twb") pod "88838224-c570-45f7-89ef-034619564318" (UID: "88838224-c570-45f7-89ef-034619564318"). InnerVolumeSpecName "kube-api-access-d7twb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:19:28.640526 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.640496    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-dshm" (OuterVolumeSpecName: "dshm") pod "88838224-c570-45f7-89ef-034619564318" (UID: "88838224-c570-45f7-89ef-034619564318"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:19:28.738573 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.738529    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88838224-c570-45f7-89ef-034619564318-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:28.738573 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.738561    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:28.738573 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.738573    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:28.738573 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.738584    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7twb\" (UniqueName: \"kubernetes.io/projected/88838224-c570-45f7-89ef-034619564318-kube-api-access-d7twb\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:28.738848 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:28.738596    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88838224-c570-45f7-89ef-034619564318-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:29.357048 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.357017    2582 generic.go:356] "Generic (PLEG): container finished" podID="88838224-c570-45f7-89ef-034619564318" containerID="39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae" exitCode=0
Jul 11 04:19:29.357240 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.357075    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"
Jul 11 04:19:29.357240 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.357124    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt" event={"ID":"88838224-c570-45f7-89ef-034619564318","Type":"ContainerDied","Data":"39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae"}
Jul 11 04:19:29.357240 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.357165    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt" event={"ID":"88838224-c570-45f7-89ef-034619564318","Type":"ContainerDied","Data":"132a0634585f8cd87b180f579cde59ef80c1d53ad02131fa93645e1aa0a43224"}
Jul 11 04:19:29.357240 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.357192    2582 scope.go:117] "RemoveContainer" containerID="39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae"
Jul 11 04:19:29.367024 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.367005    2582 scope.go:117] "RemoveContainer" containerID="39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae"
Jul 11 04:19:29.367347 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:19:29.367322    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae\": container with ID starting with 39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae not found: ID does not exist" containerID="39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae"
Jul 11 04:19:29.367456 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.367354    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae"} err="failed to get container status \"39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae\": rpc error: code = NotFound desc = could not find container \"39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae\": container with ID starting with 39d6fe6c4dc9476dbcf960b6b28789605781cc3214094f46cdd97e3f32df30ae not found: ID does not exist"
Jul 11 04:19:29.378405 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.378383    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"]
Jul 11 04:19:29.385816 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.385115    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-075325/test-model-075325-kserve-7d9cc9cc89-d46nt"]
Jul 11 04:19:29.646901 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.646817    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88838224-c570-45f7-89ef-034619564318" path="/var/lib/kubelet/pods/88838224-c570-45f7-89ef-034619564318/volumes"
Jul 11 04:19:29.647273 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:29.647231    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0de6226-b41e-41fb-8116-607ba3375e28" path="/var/lib/kubelet/pods/c0de6226-b41e-41fb-8116-607ba3375e28/volumes"
Jul 11 04:19:34.233929 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:34.233893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:34.305915 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:34.305882    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:34.809659 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:34.809628    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:19:39.238485 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.238454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:39.468190 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.468157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:39.593873 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.593771    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"]
Jul 11 04:19:39.594186 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594170    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88838224-c570-45f7-89ef-034619564318" containerName="main"
Jul 11 04:19:39.594186 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594186    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88838224-c570-45f7-89ef-034619564318" containerName="main"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594203    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0de6226-b41e-41fb-8116-607ba3375e28" containerName="main"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594209    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0de6226-b41e-41fb-8116-607ba3375e28" containerName="main"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594216    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7e8375-6159-44b8-9af1-b01a841c14cb" containerName="maas-api"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594221    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7e8375-6159-44b8-9af1-b01a841c14cb" containerName="maas-api"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594280    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0de6226-b41e-41fb-8116-607ba3375e28" containerName="main"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594287    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88838224-c570-45f7-89ef-034619564318" containerName="main"
Jul 11 04:19:39.594322 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.594296    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7e8375-6159-44b8-9af1-b01a841c14cb" containerName="maas-api"
Jul 11 04:19:39.597417 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.597393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:39.599648 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.599614    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\""
Jul 11 04:19:39.603989 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.603963    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"]
Jul 11 04:19:39.640872 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.640839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8eab0-ccf2-401a-83df-10f74f2ddc42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-075325-kzhvk\" (UID: \"6be8eab0-ccf2-401a-83df-10f74f2ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:39.742164 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.742128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8eab0-ccf2-401a-83df-10f74f2ddc42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-075325-kzhvk\" (UID: \"6be8eab0-ccf2-401a-83df-10f74f2ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:39.742781 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.742764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8eab0-ccf2-401a-83df-10f74f2ddc42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-075325-kzhvk\" (UID: \"6be8eab0-ccf2-401a-83df-10f74f2ddc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:39.910592 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:39.910490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:40.052503 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:40.052475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"]
Jul 11 04:19:40.054894 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:19:40.054860    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be8eab0_ccf2_401a_83df_10f74f2ddc42.slice/crio-3d93e72c13605cdd26032e377e5fa511d78e250219171abef03db7395895da0a WatchSource:0}: Error finding container 3d93e72c13605cdd26032e377e5fa511d78e250219171abef03db7395895da0a: Status 404 returned error can't find the container with id 3d93e72c13605cdd26032e377e5fa511d78e250219171abef03db7395895da0a
Jul 11 04:19:40.107773 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:40.107742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:40.214625 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:40.214520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:40.413906 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:40.413870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk" event={"ID":"6be8eab0-ccf2-401a-83df-10f74f2ddc42","Type":"ContainerStarted","Data":"8461bb3e4f3e5911d044b457c2ef9adcf32bf5cb0bf9969daa1974b74dad9ce5"}
Jul 11 04:19:40.414327 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:40.413929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk" event={"ID":"6be8eab0-ccf2-401a-83df-10f74f2ddc42","Type":"ContainerStarted","Data":"3d93e72c13605cdd26032e377e5fa511d78e250219171abef03db7395895da0a"}
Jul 11 04:19:40.429859 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:40.429816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk" podStartSLOduration=1.429798283 podStartE2EDuration="1.429798283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:19:40.427197331 +0000 UTC m=+2225.296643895" watchObservedRunningTime="2026-07-11 04:19:40.429798283 +0000 UTC m=+2225.299244848"
Jul 11 04:19:41.419681 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:41.419646    2582 generic.go:356] "Generic (PLEG): container finished" podID="6be8eab0-ccf2-401a-83df-10f74f2ddc42" containerID="8461bb3e4f3e5911d044b457c2ef9adcf32bf5cb0bf9969daa1974b74dad9ce5" exitCode=0
Jul 11 04:19:41.420160 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:41.419733    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk" event={"ID":"6be8eab0-ccf2-401a-83df-10f74f2ddc42","Type":"ContainerDied","Data":"8461bb3e4f3e5911d044b457c2ef9adcf32bf5cb0bf9969daa1974b74dad9ce5"}
Jul 11 04:19:42.569195 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:42.569165    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:42.670444 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:42.670413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8eab0-ccf2-401a-83df-10f74f2ddc42-maas-api-service-ca\") pod \"6be8eab0-ccf2-401a-83df-10f74f2ddc42\" (UID: \"6be8eab0-ccf2-401a-83df-10f74f2ddc42\") "
Jul 11 04:19:42.670778 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:42.670756    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be8eab0-ccf2-401a-83df-10f74f2ddc42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6be8eab0-ccf2-401a-83df-10f74f2ddc42" (UID: "6be8eab0-ccf2-401a-83df-10f74f2ddc42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:19:42.771537 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:42.771499    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8eab0-ccf2-401a-83df-10f74f2ddc42-maas-api-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:43.431242 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:43.431215    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk"
Jul 11 04:19:43.431389 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:43.431204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-075325-kzhvk" event={"ID":"6be8eab0-ccf2-401a-83df-10f74f2ddc42","Type":"ContainerDied","Data":"3d93e72c13605cdd26032e377e5fa511d78e250219171abef03db7395895da0a"}
Jul 11 04:19:43.431389 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:43.431342    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d93e72c13605cdd26032e377e5fa511d78e250219171abef03db7395895da0a"
Jul 11 04:19:49.975072 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:49.974976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"]
Jul 11 04:19:49.975600 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:49.975572    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" podUID="97c27d51-f696-4d2c-a456-3478bde9112f" containerName="maas-api" containerID="cri-o://79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940" gracePeriod=30
Jul 11 04:19:49.998286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:49.998260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:19:50.236172 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.236151    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:19:50.337133 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.337065    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c27d51-f696-4d2c-a456-3478bde9112f-openshift-service-ca\") pod \"97c27d51-f696-4d2c-a456-3478bde9112f\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") "
Jul 11 04:19:50.337336 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.337151    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hj52\" (UniqueName: \"kubernetes.io/projected/97c27d51-f696-4d2c-a456-3478bde9112f-kube-api-access-8hj52\") pod \"97c27d51-f696-4d2c-a456-3478bde9112f\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") "
Jul 11 04:19:50.337336 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.337264    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") pod \"97c27d51-f696-4d2c-a456-3478bde9112f\" (UID: \"97c27d51-f696-4d2c-a456-3478bde9112f\") "
Jul 11 04:19:50.337470 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.337447    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c27d51-f696-4d2c-a456-3478bde9112f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97c27d51-f696-4d2c-a456-3478bde9112f" (UID: "97c27d51-f696-4d2c-a456-3478bde9112f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:19:50.337565 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.337550    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c27d51-f696-4d2c-a456-3478bde9112f-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:50.339300 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.339274    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c27d51-f696-4d2c-a456-3478bde9112f-kube-api-access-8hj52" (OuterVolumeSpecName: "kube-api-access-8hj52") pod "97c27d51-f696-4d2c-a456-3478bde9112f" (UID: "97c27d51-f696-4d2c-a456-3478bde9112f"). InnerVolumeSpecName "kube-api-access-8hj52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:19:50.339454 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.339424    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97c27d51-f696-4d2c-a456-3478bde9112f" (UID: "97c27d51-f696-4d2c-a456-3478bde9112f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:19:50.438797 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.438753    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c27d51-f696-4d2c-a456-3478bde9112f-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:50.438797 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.438791    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hj52\" (UniqueName: \"kubernetes.io/projected/97c27d51-f696-4d2c-a456-3478bde9112f-kube-api-access-8hj52\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:19:50.469825 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.469788    2582 generic.go:356] "Generic (PLEG): container finished" podID="97c27d51-f696-4d2c-a456-3478bde9112f" containerID="79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940" exitCode=0
Jul 11 04:19:50.470074 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.469838    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"
Jul 11 04:19:50.470074 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.469876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" event={"ID":"97c27d51-f696-4d2c-a456-3478bde9112f","Type":"ContainerDied","Data":"79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940"}
Jul 11 04:19:50.470074 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.469926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq" event={"ID":"97c27d51-f696-4d2c-a456-3478bde9112f","Type":"ContainerDied","Data":"6bef2777b2d789663c850723ce010a350345b0bf22f767eb967f142a750a6059"}
Jul 11 04:19:50.470074 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.469950    2582 scope.go:117] "RemoveContainer" containerID="79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940"
Jul 11 04:19:50.478895 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.478878    2582 scope.go:117] "RemoveContainer" containerID="79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940"
Jul 11 04:19:50.479183 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:19:50.479166    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940\": container with ID starting with 79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940 not found: ID does not exist" containerID="79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940"
Jul 11 04:19:50.479233 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.479191    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940"} err="failed to get container status \"79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940\": rpc error: code = NotFound desc = could not find container \"79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940\": container with ID starting with 79997762b9eebaa3671444cd83c6fde27a5c7091ea0142ae1009791ff9ba6940 not found: ID does not exist"
Jul 11 04:19:50.490992 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.490935    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"]
Jul 11 04:19:50.496571 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:50.496548    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-075325-78c47cd6dc-jj8pq"]
Jul 11 04:19:51.646125 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:51.646075    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c27d51-f696-4d2c-a456-3478bde9112f" path="/var/lib/kubelet/pods/97c27d51-f696-4d2c-a456-3478bde9112f/volumes"
Jul 11 04:19:53.205422 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:53.205391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:19:54.675715 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:19:54.675674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:00.379024 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:00.378941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:00.435160 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:00.435129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:03.749043 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:03.749007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:05.410757 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:05.410722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:05.671174 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:05.671061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:06.017437 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:06.017403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:06.339805 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:06.339728    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:07.499027 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:07.498987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:09.706007 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.705965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:09.814872 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.814843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:09.901843 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.901809    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"]
Jul 11 04:20:09.902295 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.902278    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c27d51-f696-4d2c-a456-3478bde9112f" containerName="maas-api"
Jul 11 04:20:09.902395 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.902297    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c27d51-f696-4d2c-a456-3478bde9112f" containerName="maas-api"
Jul 11 04:20:09.902395 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.902341    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be8eab0-ccf2-401a-83df-10f74f2ddc42" containerName="revoke-keys"
Jul 11 04:20:09.902395 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.902352    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be8eab0-ccf2-401a-83df-10f74f2ddc42" containerName="revoke-keys"
Jul 11 04:20:09.902554 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.902446    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c27d51-f696-4d2c-a456-3478bde9112f" containerName="maas-api"
Jul 11 04:20:09.902554 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.902462    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be8eab0-ccf2-401a-83df-10f74f2ddc42" containerName="revoke-keys"
Jul 11 04:20:09.905565 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.905534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:09.907759 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.907738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-20f328-serving-cert\""
Jul 11 04:20:09.915303 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.915280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"]
Jul 11 04:20:09.936080 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:09.936056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:10.007616 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.007576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gjt\" (UniqueName: \"kubernetes.io/projected/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-kube-api-access-r7gjt\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.007794 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.007642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-openshift-service-ca\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.007794 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.007692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-maas-api-tls\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.035409 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.035376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:10.108573 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.108540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-openshift-service-ca\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.108760 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.108590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-maas-api-tls\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.108760 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.108664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gjt\" (UniqueName: \"kubernetes.io/projected/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-kube-api-access-r7gjt\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.109251 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.109230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-openshift-service-ca\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.111163 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.111139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-maas-api-tls\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.118671 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.118650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gjt\" (UniqueName: \"kubernetes.io/projected/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-kube-api-access-r7gjt\") pod \"maas-api-e2e-api-a-20f328-96b4f689d-xxxls\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.216652 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.216620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:10.363669 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.363577    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"]
Jul 11 04:20:10.365886 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:20:10.365860    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c6a7d5_80e4_49fe_a79f_89f2e1a83ed0.slice/crio-0b90b19fc41519a837db2605499298f3a92d84a25c41291fb5ebee208035c432 WatchSource:0}: Error finding container 0b90b19fc41519a837db2605499298f3a92d84a25c41291fb5ebee208035c432: Status 404 returned error can't find the container with id 0b90b19fc41519a837db2605499298f3a92d84a25c41291fb5ebee208035c432
Jul 11 04:20:10.367414 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.367394    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 11 04:20:10.371247 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.371220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:10.471991 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.471952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:10.569226 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:10.569133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls" event={"ID":"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0","Type":"ContainerStarted","Data":"0b90b19fc41519a837db2605499298f3a92d84a25c41291fb5ebee208035c432"}
Jul 11 04:20:11.575617 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:11.575587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls" event={"ID":"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0","Type":"ContainerStarted","Data":"4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877"}
Jul 11 04:20:11.576012 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:11.575654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:11.591286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:11.591241    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls" podStartSLOduration=2.109922655 podStartE2EDuration="2.591220656s" totalImagesPullingTime="481.298001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:20:11.589911365 +0000 UTC m=+2256.459357929" watchObservedRunningTime="2026-07-11 04:20:11.591220656 +0000 UTC m=+2256.460667220"
Jul 11 04:20:13.809362 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:13.809320    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:20:14.213938 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:14.213841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:14.678384 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:14.678342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:14.989886 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:14.989861    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:15.287085 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:15.287008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:17.586330 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.586306    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:17.753371 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.753330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:17.850961 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.850853    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:17.916644 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.916612    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"]
Jul 11 04:20:17.920490 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.920466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:17.922841 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.922820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e88abb-serving-cert\""
Jul 11 04:20:17.929056 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.929018    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"]
Jul 11 04:20:17.951964 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.951930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:17.983425 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.983398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwwf\" (UniqueName: \"kubernetes.io/projected/e0a1b966-d382-46c5-b913-9cd39fc05cc5-kube-api-access-fwwwf\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:17.983589 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.983479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1b966-d382-46c5-b913-9cd39fc05cc5-openshift-service-ca\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:17.983589 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:17.983546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.084579 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.084546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1b966-d382-46c5-b913-9cd39fc05cc5-openshift-service-ca\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.084748 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.084606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.084748 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.084650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwwf\" (UniqueName: \"kubernetes.io/projected/e0a1b966-d382-46c5-b913-9cd39fc05cc5-kube-api-access-fwwwf\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.084823 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:20:18.084777    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-serving-cert: secret "maas-api-e2e-api-b-e88abb-serving-cert" not found
Jul 11 04:20:18.084857 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:20:18.084851    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls podName:e0a1b966-d382-46c5-b913-9cd39fc05cc5 nodeName:}" failed. No retries permitted until 2026-07-11 04:20:18.584831565 +0000 UTC m=+2263.454278114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls") pod "maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" (UID: "e0a1b966-d382-46c5-b913-9cd39fc05cc5") : secret "maas-api-e2e-api-b-e88abb-serving-cert" not found
Jul 11 04:20:18.085199 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.085181    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1b966-d382-46c5-b913-9cd39fc05cc5-openshift-service-ca\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.092910 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.092875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwwf\" (UniqueName: \"kubernetes.io/projected/e0a1b966-d382-46c5-b913-9cd39fc05cc5-kube-api-access-fwwwf\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.175311 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.175214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:18.387306 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.387272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:18.493010 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.492977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:18.588422 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.588391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.591188 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.591163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls\") pod \"maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.834944 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.834852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:18.984448 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:18.984418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"]
Jul 11 04:20:18.986457 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:20:18.986415    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a1b966_d382_46c5_b913_9cd39fc05cc5.slice/crio-e570c9623e085290aea14c921cdcc39c35b6da9838e75e8ef3a0cf596af8f906 WatchSource:0}: Error finding container e570c9623e085290aea14c921cdcc39c35b6da9838e75e8ef3a0cf596af8f906: Status 404 returned error can't find the container with id e570c9623e085290aea14c921cdcc39c35b6da9838e75e8ef3a0cf596af8f906
Jul 11 04:20:19.620539 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:19.620502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" event={"ID":"e0a1b966-d382-46c5-b913-9cd39fc05cc5","Type":"ContainerStarted","Data":"6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2"}
Jul 11 04:20:19.620981 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:19.620701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:19.620981 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:19.620728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" event={"ID":"e0a1b966-d382-46c5-b913-9cd39fc05cc5","Type":"ContainerStarted","Data":"e570c9623e085290aea14c921cdcc39c35b6da9838e75e8ef3a0cf596af8f906"}
Jul 11 04:20:19.636949 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:19.636851    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" podStartSLOduration=2.24879697 podStartE2EDuration="2.636834712s" totalImagesPullingTime="388.037742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:20:19.63627508 +0000 UTC m=+2264.505721646" watchObservedRunningTime="2026-07-11 04:20:19.636834712 +0000 UTC m=+2264.506281278"
Jul 11 04:20:22.251407 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:22.251363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:25.631867 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:25.631828    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:20:26.719077 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:26.719040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:20:30.314530 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:30.314492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:30.385964 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:30.385935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:31.021766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:31.021736    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:31.123584 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:31.123547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:39.225231 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:39.225197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:39.326319 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:39.326285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:40.892867 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:40.892831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:40.994166 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:40.994131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:41.121866 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.121830    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"]
Jul 11 04:20:41.122143 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.122074    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls" podUID="a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" containerName="maas-api" containerID="cri-o://4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877" gracePeriod=30
Jul 11 04:20:41.149763 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.149695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:41.387254 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.387231    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:41.511198 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.511169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-openshift-service-ca\") pod \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") "
Jul 11 04:20:41.511369 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.511255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-maas-api-tls\") pod \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") "
Jul 11 04:20:41.511369 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.511306    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7gjt\" (UniqueName: \"kubernetes.io/projected/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-kube-api-access-r7gjt\") pod \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\" (UID: \"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0\") "
Jul 11 04:20:41.511621 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.511599    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" (UID: "a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:20:41.513506 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.513474    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-kube-api-access-r7gjt" (OuterVolumeSpecName: "kube-api-access-r7gjt") pod "a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" (UID: "a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0"). InnerVolumeSpecName "kube-api-access-r7gjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:20:41.513600 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.513559    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" (UID: "a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:20:41.612069 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.612034    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:20:41.612069 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.612063    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:20:41.612069 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.612074    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gjt\" (UniqueName: \"kubernetes.io/projected/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0-kube-api-access-r7gjt\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:20:41.731057 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.731023    2582 generic.go:356] "Generic (PLEG): container finished" podID="a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" containerID="4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877" exitCode=0
Jul 11 04:20:41.731260 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.731070    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"
Jul 11 04:20:41.731260 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.731127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls" event={"ID":"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0","Type":"ContainerDied","Data":"4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877"}
Jul 11 04:20:41.731260 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.731179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls" event={"ID":"a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0","Type":"ContainerDied","Data":"0b90b19fc41519a837db2605499298f3a92d84a25c41291fb5ebee208035c432"}
Jul 11 04:20:41.731260 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.731209    2582 scope.go:117] "RemoveContainer" containerID="4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877"
Jul 11 04:20:41.739878 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.739860    2582 scope.go:117] "RemoveContainer" containerID="4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877"
Jul 11 04:20:41.740146 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:20:41.740123    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877\": container with ID starting with 4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877 not found: ID does not exist" containerID="4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877"
Jul 11 04:20:41.740226 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.740155    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877"} err="failed to get container status \"4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877\": rpc error: code = NotFound desc = could not find container \"4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877\": container with ID starting with 4fb544eb01badf23f9668eaa5df21bc1f029a6d9ecc0934a62f12730d8e90877 not found: ID does not exist"
Jul 11 04:20:41.747305 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.747284    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"]
Jul 11 04:20:41.750540 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:41.750518    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-20f328-96b4f689d-xxxls"]
Jul 11 04:20:43.646191 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:43.646159    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" path="/var/lib/kubelet/pods/a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0/volumes"
Jul 11 04:20:44.706113 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:44.706070    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:20:45.474204 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:45.474168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:51.348159 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:51.348125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:51.449872 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:51.449825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:56.166395 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.166350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:56.395724 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.395681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:56.533037 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.533004    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"]
Jul 11 04:20:56.533434 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.533420    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" containerName="maas-api"
Jul 11 04:20:56.533478 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.533435    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" containerName="maas-api"
Jul 11 04:20:56.533520 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.533511    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c6a7d5-80e4-49fe-a79f-89f2e1a83ed0" containerName="maas-api"
Jul 11 04:20:56.538036 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.538021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:56.539411 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.539389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f19ffa-cf06-46de-bcb9-0259af760f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e88abb-8n68t\" (UID: \"48f19ffa-cf06-46de-bcb9-0259af760f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:56.540613 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.540556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\""
Jul 11 04:20:56.542953 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.542924    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"]
Jul 11 04:20:56.640554 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.640523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f19ffa-cf06-46de-bcb9-0259af760f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e88abb-8n68t\" (UID: \"48f19ffa-cf06-46de-bcb9-0259af760f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:56.641165 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.641144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f19ffa-cf06-46de-bcb9-0259af760f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e88abb-8n68t\" (UID: \"48f19ffa-cf06-46de-bcb9-0259af760f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:56.851286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.851184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:56.989024 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:56.989003    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"]
Jul 11 04:20:56.990969 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:20:56.990943    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f19ffa_cf06_46de_bcb9_0259af760f30.slice/crio-5f0ee3fb59755c5b07290b8361901f753cf7a35578f4c215699f042fa3be5003 WatchSource:0}: Error finding container 5f0ee3fb59755c5b07290b8361901f753cf7a35578f4c215699f042fa3be5003: Status 404 returned error can't find the container with id 5f0ee3fb59755c5b07290b8361901f753cf7a35578f4c215699f042fa3be5003
Jul 11 04:20:57.170696 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:57.170672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:57.271835 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:57.271806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:20:57.814265 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:57.814231    2582 generic.go:356] "Generic (PLEG): container finished" podID="48f19ffa-cf06-46de-bcb9-0259af760f30" containerID="e683132a9ad6d2eea07992b60e99575640e95d8f08fa39f4c4b69f14ba9c7a02" exitCode=0
Jul 11 04:20:57.814431 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:57.814272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t" event={"ID":"48f19ffa-cf06-46de-bcb9-0259af760f30","Type":"ContainerDied","Data":"e683132a9ad6d2eea07992b60e99575640e95d8f08fa39f4c4b69f14ba9c7a02"}
Jul 11 04:20:57.814431 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:57.814311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t" event={"ID":"48f19ffa-cf06-46de-bcb9-0259af760f30","Type":"ContainerStarted","Data":"5f0ee3fb59755c5b07290b8361901f753cf7a35578f4c215699f042fa3be5003"}
Jul 11 04:20:58.955995 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:58.955961    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:59.058716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:59.058686    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f19ffa-cf06-46de-bcb9-0259af760f30-maas-api-service-ca\") pod \"48f19ffa-cf06-46de-bcb9-0259af760f30\" (UID: \"48f19ffa-cf06-46de-bcb9-0259af760f30\") "
Jul 11 04:20:59.059049 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:59.059025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f19ffa-cf06-46de-bcb9-0259af760f30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48f19ffa-cf06-46de-bcb9-0259af760f30" (UID: "48f19ffa-cf06-46de-bcb9-0259af760f30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:20:59.159618 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:59.159529    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f19ffa-cf06-46de-bcb9-0259af760f30-maas-api-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:20:59.824950 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:59.824922    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t"
Jul 11 04:20:59.825147 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:59.824912    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e88abb-8n68t" event={"ID":"48f19ffa-cf06-46de-bcb9-0259af760f30","Type":"ContainerDied","Data":"5f0ee3fb59755c5b07290b8361901f753cf7a35578f4c215699f042fa3be5003"}
Jul 11 04:20:59.825147 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:20:59.825050    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0ee3fb59755c5b07290b8361901f753cf7a35578f4c215699f042fa3be5003"
Jul 11 04:21:06.901724 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:06.901677    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"]
Jul 11 04:21:06.902220 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:06.902145    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" podUID="e0a1b966-d382-46c5-b913-9cd39fc05cc5" containerName="maas-api" containerID="cri-o://6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2" gracePeriod=30
Jul 11 04:21:06.927450 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:06.927425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:07.163663 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.163584    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:21:07.230481 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.230447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls\") pod \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") "
Jul 11 04:21:07.230481 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.230481    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1b966-d382-46c5-b913-9cd39fc05cc5-openshift-service-ca\") pod \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") "
Jul 11 04:21:07.230678 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.230570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwwwf\" (UniqueName: \"kubernetes.io/projected/e0a1b966-d382-46c5-b913-9cd39fc05cc5-kube-api-access-fwwwf\") pod \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\" (UID: \"e0a1b966-d382-46c5-b913-9cd39fc05cc5\") "
Jul 11 04:21:07.230900 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.230877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a1b966-d382-46c5-b913-9cd39fc05cc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0a1b966-d382-46c5-b913-9cd39fc05cc5" (UID: "e0a1b966-d382-46c5-b913-9cd39fc05cc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:21:07.232759 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.232735    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a1b966-d382-46c5-b913-9cd39fc05cc5-kube-api-access-fwwwf" (OuterVolumeSpecName: "kube-api-access-fwwwf") pod "e0a1b966-d382-46c5-b913-9cd39fc05cc5" (UID: "e0a1b966-d382-46c5-b913-9cd39fc05cc5"). InnerVolumeSpecName "kube-api-access-fwwwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:21:07.232824 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.232801    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0a1b966-d382-46c5-b913-9cd39fc05cc5" (UID: "e0a1b966-d382-46c5-b913-9cd39fc05cc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:21:07.332215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.332174    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a1b966-d382-46c5-b913-9cd39fc05cc5-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:21:07.332215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.332211    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1b966-d382-46c5-b913-9cd39fc05cc5-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:21:07.332422 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.332228    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwwf\" (UniqueName: \"kubernetes.io/projected/e0a1b966-d382-46c5-b913-9cd39fc05cc5-kube-api-access-fwwwf\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:21:07.866878 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.866843    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0a1b966-d382-46c5-b913-9cd39fc05cc5" containerID="6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2" exitCode=0
Jul 11 04:21:07.867053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.866896    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"
Jul 11 04:21:07.867053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.866922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" event={"ID":"e0a1b966-d382-46c5-b913-9cd39fc05cc5","Type":"ContainerDied","Data":"6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2"}
Jul 11 04:21:07.867053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.866976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv" event={"ID":"e0a1b966-d382-46c5-b913-9cd39fc05cc5","Type":"ContainerDied","Data":"e570c9623e085290aea14c921cdcc39c35b6da9838e75e8ef3a0cf596af8f906"}
Jul 11 04:21:07.867053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.867032    2582 scope.go:117] "RemoveContainer" containerID="6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2"
Jul 11 04:21:07.876140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.876123    2582 scope.go:117] "RemoveContainer" containerID="6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2"
Jul 11 04:21:07.876403 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:21:07.876385    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2\": container with ID starting with 6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2 not found: ID does not exist" containerID="6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2"
Jul 11 04:21:07.876449 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.876412    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2"} err="failed to get container status \"6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2\": rpc error: code = NotFound desc = could not find container \"6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2\": container with ID starting with 6dc0fe2dbf3b574952abfaaf93cb198d782a4039ac47f0ae528d432244f0eda2 not found: ID does not exist"
Jul 11 04:21:07.881856 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.881756    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"]
Jul 11 04:21:07.883702 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:07.883679    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e88abb-54bc9564d-5nwtv"]
Jul 11 04:21:09.645421 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:09.645378    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a1b966-d382-46c5-b913-9cd39fc05cc5" path="/var/lib/kubelet/pods/e0a1b966-d382-46c5-b913-9cd39fc05cc5/volumes"
Jul 11 04:21:10.101504 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:10.101470    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:21:11.607882 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:11.607845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:17.396807 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:17.396776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:17.452877 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:17.452848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:22.321923 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:22.321886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:22.556199 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:22.556162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:22.924394 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:22.924364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:23.251158 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:23.251126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:23.518870 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:23.518789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.120897 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.120859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.197576 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.197537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.205271 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205244    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"]
Jul 11 04:21:26.205617 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f19ffa-cf06-46de-bcb9-0259af760f30" containerName="revoke-keys"
Jul 11 04:21:26.205666 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205619    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f19ffa-cf06-46de-bcb9-0259af760f30" containerName="revoke-keys"
Jul 11 04:21:26.205666 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205632    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a1b966-d382-46c5-b913-9cd39fc05cc5" containerName="maas-api"
Jul 11 04:21:26.205666 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205638    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a1b966-d382-46c5-b913-9cd39fc05cc5" containerName="maas-api"
Jul 11 04:21:26.205769 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205706    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f19ffa-cf06-46de-bcb9-0259af760f30" containerName="revoke-keys"
Jul 11 04:21:26.205769 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.205714    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a1b966-d382-46c5-b913-9cd39fc05cc5" containerName="maas-api"
Jul 11 04:21:26.208925 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.208899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.211469 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.211449    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6ee78d-serving-cert\""
Jul 11 04:21:26.223999 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.223972    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"]
Jul 11 04:21:26.268946 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.268908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.317343 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.317308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drq5v\" (UniqueName: \"kubernetes.io/projected/2446db14-eb6e-4176-b3d2-344090fe9fee-kube-api-access-drq5v\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.317531 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.317351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2446db14-eb6e-4176-b3d2-344090fe9fee-maas-api-tls\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.317531 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.317495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2446db14-eb6e-4176-b3d2-344090fe9fee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.418291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.418183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drq5v\" (UniqueName: \"kubernetes.io/projected/2446db14-eb6e-4176-b3d2-344090fe9fee-kube-api-access-drq5v\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.418291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.418222    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2446db14-eb6e-4176-b3d2-344090fe9fee-maas-api-tls\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.418291 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.418281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2446db14-eb6e-4176-b3d2-344090fe9fee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.418849 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.418828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2446db14-eb6e-4176-b3d2-344090fe9fee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.421050 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.421023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2446db14-eb6e-4176-b3d2-344090fe9fee-maas-api-tls\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.422218 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.422193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.426869 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.426848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drq5v\" (UniqueName: \"kubernetes.io/projected/2446db14-eb6e-4176-b3d2-344090fe9fee-kube-api-access-drq5v\") pod \"maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.522410 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.522380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:26.664569 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.664546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"]
Jul 11 04:21:26.667001 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:21:26.666974    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2446db14_eb6e_4176_b3d2_344090fe9fee.slice/crio-317b9426dd135834fccc902dd55ce610e0c15f50a8ecb61453a5b0efbbad1183 WatchSource:0}: Error finding container 317b9426dd135834fccc902dd55ce610e0c15f50a8ecb61453a5b0efbbad1183: Status 404 returned error can't find the container with id 317b9426dd135834fccc902dd55ce610e0c15f50a8ecb61453a5b0efbbad1183
Jul 11 04:21:26.681592 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.681569    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.782793 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.782757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:26.963617 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:26.963528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44" event={"ID":"2446db14-eb6e-4176-b3d2-344090fe9fee","Type":"ContainerStarted","Data":"317b9426dd135834fccc902dd55ce610e0c15f50a8ecb61453a5b0efbbad1183"}
Jul 11 04:21:27.969778 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:27.969742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44" event={"ID":"2446db14-eb6e-4176-b3d2-344090fe9fee","Type":"ContainerStarted","Data":"ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad"}
Jul 11 04:21:27.970450 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:27.969822    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:27.990832 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:27.990781    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44" podStartSLOduration=1.596444934 podStartE2EDuration="1.990764005s" totalImagesPullingTime="394.319071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:21:27.98990578 +0000 UTC m=+2332.859352343" watchObservedRunningTime="2026-07-11 04:21:27.990764005 +0000 UTC m=+2332.860210570"
Jul 11 04:21:30.661593 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:30.661561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:30.946171 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:30.946083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:21:31.016581 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:31.016536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:31.322951 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:31.322897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:31.592282 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:31.592168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:33.980867 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:33.980833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:21:34.146289 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.146257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:34.247580 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.247546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:34.321817 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.321781    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"]
Jul 11 04:21:34.325820 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.325794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.328304 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.328281    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1da175-serving-cert\""
Jul 11 04:21:34.335437 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.335410    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"]
Jul 11 04:21:34.358408 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.358375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:34.492004 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.491968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.492211 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.492012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f744fe7-7e56-41c8-8704-0b829ddb2044-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.492258 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.492216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77s8\" (UniqueName: \"kubernetes.io/projected/4f744fe7-7e56-41c8-8704-0b829ddb2044-kube-api-access-p77s8\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.593684 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.593600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77s8\" (UniqueName: \"kubernetes.io/projected/4f744fe7-7e56-41c8-8704-0b829ddb2044-kube-api-access-p77s8\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.593684 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.593642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.593684 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.593666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f744fe7-7e56-41c8-8704-0b829ddb2044-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.593903 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:21:34.593769    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-serving-cert: secret "maas-api-e2e-shared-b-1da175-serving-cert" not found
Jul 11 04:21:34.593903 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:21:34.593845    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls podName:4f744fe7-7e56-41c8-8704-0b829ddb2044 nodeName:}" failed. No retries permitted until 2026-07-11 04:21:35.093824285 +0000 UTC m=+2339.963270830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls") pod "maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" (UID: "4f744fe7-7e56-41c8-8704-0b829ddb2044") : secret "maas-api-e2e-shared-b-1da175-serving-cert" not found
Jul 11 04:21:34.594269 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.594251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f744fe7-7e56-41c8-8704-0b829ddb2044-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.601870 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.601837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77s8\" (UniqueName: \"kubernetes.io/projected/4f744fe7-7e56-41c8-8704-0b829ddb2044-kube-api-access-p77s8\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:34.895475 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.895392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:34.992015 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:34.991978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:35.096742 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:35.096712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:35.099454 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:35.099431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls\") pod \"maas-api-e2e-shared-b-1da175-8475b7c954-f55jj\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:35.237966 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:35.237927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:35.383402 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:35.383376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"]
Jul 11 04:21:35.385785 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:21:35.385758    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f744fe7_7e56_41c8_8704_0b829ddb2044.slice/crio-773721436dc553fa2074a10d405545ef16abae1b8fb2a3a3b41a501059a2802d WatchSource:0}: Error finding container 773721436dc553fa2074a10d405545ef16abae1b8fb2a3a3b41a501059a2802d: Status 404 returned error can't find the container with id 773721436dc553fa2074a10d405545ef16abae1b8fb2a3a3b41a501059a2802d
Jul 11 04:21:36.016427 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:36.016389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" event={"ID":"4f744fe7-7e56-41c8-8704-0b829ddb2044","Type":"ContainerStarted","Data":"f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b"}
Jul 11 04:21:36.016817 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:36.016467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" event={"ID":"4f744fe7-7e56-41c8-8704-0b829ddb2044","Type":"ContainerStarted","Data":"773721436dc553fa2074a10d405545ef16abae1b8fb2a3a3b41a501059a2802d"}
Jul 11 04:21:36.016817 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:36.016586    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:36.033703 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:36.033654    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" podStartSLOduration=1.692564171 podStartE2EDuration="2.033639669s" totalImagesPullingTime="341.075498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:21:36.032676529 +0000 UTC m=+2340.902123093" watchObservedRunningTime="2026-07-11 04:21:36.033639669 +0000 UTC m=+2340.903086253"
Jul 11 04:21:38.651629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:38.651560    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:42.005919 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:42.005889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:21:42.026352 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:42.026328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:21:42.303224 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:42.303140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:47.397578 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:47.397547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:47.496538 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:47.496503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:50.013884 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.013851    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"]
Jul 11 04:21:50.017493 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.017476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.021069 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.021039    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6ee78d\"/\"default-dockercfg-2dzlv\""
Jul 11 04:21:50.021069 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.021041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6ee78d\"/\"openshift-service-ca.crt\""
Jul 11 04:21:50.021296 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.021041    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6ee78d\"/\"auth-test-model-6ee78d-kserve-self-signed-certs\""
Jul 11 04:21:50.021296 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.021178    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6ee78d\"/\"kube-root-ca.crt\""
Jul 11 04:21:50.025494 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025470    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"]
Jul 11 04:21:50.025599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-home\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.025599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-tmp-dir\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.025599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fc479e-038f-41c4-934b-84666af325f3-tls-certs\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.025729 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-model-cache\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.025729 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jcs\" (UniqueName: \"kubernetes.io/projected/f9fc479e-038f-41c4-934b-84666af325f3-kube-api-access-h2jcs\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.025831 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.025794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-dshm\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.035865 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.035835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:50.126448 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-home\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-tmp-dir\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fc479e-038f-41c4-934b-84666af325f3-tls-certs\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-model-cache\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jcs\" (UniqueName: \"kubernetes.io/projected/f9fc479e-038f-41c4-934b-84666af325f3-kube-api-access-h2jcs\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-dshm\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126895 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-home\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126938 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-tmp-dir\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.126938 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.126910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-model-cache\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.128965 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.128945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-dshm\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.129264 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.129246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fc479e-038f-41c4-934b-84666af325f3-tls-certs\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.133998 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.133979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jcs\" (UniqueName: \"kubernetes.io/projected/f9fc479e-038f-41c4-934b-84666af325f3-kube-api-access-h2jcs\") pod \"auth-test-model-6ee78d-kserve-5d956976c-82jn6\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") " pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.328370 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.328269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:21:50.475652 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:50.475625    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"]
Jul 11 04:21:50.478067 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:21:50.478036    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9fc479e_038f_41c4_934b_84666af325f3.slice/crio-4e86742df388db0df131dbb33649af0a75bdd5fc5e7dab5a9f28e9d1305015b4 WatchSource:0}: Error finding container 4e86742df388db0df131dbb33649af0a75bdd5fc5e7dab5a9f28e9d1305015b4: Status 404 returned error can't find the container with id 4e86742df388db0df131dbb33649af0a75bdd5fc5e7dab5a9f28e9d1305015b4
Jul 11 04:21:51.091817 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:51.091782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6" event={"ID":"f9fc479e-038f-41c4-934b-84666af325f3","Type":"ContainerStarted","Data":"74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab"}
Jul 11 04:21:51.091817 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:51.091830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6" event={"ID":"f9fc479e-038f-41c4-934b-84666af325f3","Type":"ContainerStarted","Data":"4e86742df388db0df131dbb33649af0a75bdd5fc5e7dab5a9f28e9d1305015b4"}
Jul 11 04:21:51.108582 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:51.108540    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6" podStartSLOduration=2.108523894 podStartE2EDuration="2.108523894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:21:51.107169998 +0000 UTC m=+2355.976616564" watchObservedRunningTime="2026-07-11 04:21:51.108523894 +0000 UTC m=+2355.977970457"
Jul 11 04:21:55.299064 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:55.299033    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:21:55.604375 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:55.604286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:21:56.762539 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:21:56.762507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:22:00.328369 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:00.328331    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:22:00.328774 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:00.328384    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:22:00.341088 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:00.341056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:22:01.162445 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:01.162418    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:22:07.812782 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.812747    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"]
Jul 11 04:22:07.816941 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.816921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.820470 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.820432    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"default-dockercfg-5slww\""
Jul 11 04:22:07.820470 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.820458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"auth-test-model-1da175-kserve-self-signed-certs\""
Jul 11 04:22:07.820470 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.820436    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"kube-root-ca.crt\""
Jul 11 04:22:07.820843 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.820738    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"openshift-service-ca.crt\""
Jul 11 04:22:07.824148 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.824123    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"]
Jul 11 04:22:07.841091 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.841068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:22:07.896162 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.896123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/7884daa9-4fb8-498f-9602-9db9c5e485a7-kube-api-access-drf8t\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.896347 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.896186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-tmp-dir\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.896347 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.896239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-model-cache\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.896347 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.896299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7884daa9-4fb8-498f-9602-9db9c5e485a7-tls-certs\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.896347 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.896343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-home\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.896542 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.896413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-dshm\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.997790 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.997758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7884daa9-4fb8-498f-9602-9db9c5e485a7-tls-certs\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.997966 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.997804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-home\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.997966 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.997838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-dshm\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.998071 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.997966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/7884daa9-4fb8-498f-9602-9db9c5e485a7-kube-api-access-drf8t\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.998071 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.998025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-tmp-dir\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.998207 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.998073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-model-cache\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.998370 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.998339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-home\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.998370 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.998357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-tmp-dir\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:07.998519 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:07.998425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-model-cache\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:08.000293 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:08.000276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-dshm\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:08.000406 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:08.000389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7884daa9-4fb8-498f-9602-9db9c5e485a7-tls-certs\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:08.005240 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:08.005222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/7884daa9-4fb8-498f-9602-9db9c5e485a7-kube-api-access-drf8t\") pod \"auth-test-model-1da175-kserve-57677f8b87-zc7kx\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") " pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:08.129343 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:08.129250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:08.273397 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:08.273339    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"]
Jul 11 04:22:08.275676 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:22:08.275648    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7884daa9_4fb8_498f_9602_9db9c5e485a7.slice/crio-ad64698cdc9aed8a89c8e36ab14f48b90a625becf328be721418fbc153be35c4 WatchSource:0}: Error finding container ad64698cdc9aed8a89c8e36ab14f48b90a625becf328be721418fbc153be35c4: Status 404 returned error can't find the container with id ad64698cdc9aed8a89c8e36ab14f48b90a625becf328be721418fbc153be35c4
Jul 11 04:22:09.194793 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:09.194755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" event={"ID":"7884daa9-4fb8-498f-9602-9db9c5e485a7","Type":"ContainerStarted","Data":"ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec"}
Jul 11 04:22:09.195227 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:09.194806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" event={"ID":"7884daa9-4fb8-498f-9602-9db9c5e485a7","Type":"ContainerStarted","Data":"ad64698cdc9aed8a89c8e36ab14f48b90a625becf328be721418fbc153be35c4"}
Jul 11 04:22:09.211733 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:09.211687    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" podStartSLOduration=2.211671972 podStartE2EDuration="2.211671972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:22:09.210975284 +0000 UTC m=+2374.080421912" watchObservedRunningTime="2026-07-11 04:22:09.211671972 +0000 UTC m=+2374.081118536"
Jul 11 04:22:11.509169 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:11.509133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:22:18.129447 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:18.129414    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:18.129942 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:18.129473    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:18.141886 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:18.141859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:18.259883 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:18.259848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:22:35.897820 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:35.897786    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:22:35.906116 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:35.906077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:22:42.906207 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:22:42.906176    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:23:00.612858 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:23:00.612767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:23:18.517772 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:23:18.517735    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:23:35.210188 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:23:35.210153    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:23:53.911946 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:23:53.911912    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:24:10.906005 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:10.905966    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:24:27.706278 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:27.706182    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:24:31.985053 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:31.985020    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"]
Jul 11 04:24:32.030606 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:32.030565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:24:32.983438 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:32.983403    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"]
Jul 11 04:24:33.004741 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:33.004708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:24:41.328904 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:41.328868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:24:47.005812 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.005764    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6" podUID="f9fc479e-038f-41c4-934b-84666af325f3" containerName="main" containerID="cri-o://74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab" gracePeriod=45
Jul 11 04:24:47.269580 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.269516    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:24:47.290862 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.290836    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-dshm\") pod \"f9fc479e-038f-41c4-934b-84666af325f3\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") "
Jul 11 04:24:47.291040 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.290874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-model-cache\") pod \"f9fc479e-038f-41c4-934b-84666af325f3\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") "
Jul 11 04:24:47.291040 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.290907    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-home\") pod \"f9fc479e-038f-41c4-934b-84666af325f3\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") "
Jul 11 04:24:47.291040 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.290960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-tmp-dir\") pod \"f9fc479e-038f-41c4-934b-84666af325f3\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") "
Jul 11 04:24:47.291040 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.290993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2jcs\" (UniqueName: \"kubernetes.io/projected/f9fc479e-038f-41c4-934b-84666af325f3-kube-api-access-h2jcs\") pod \"f9fc479e-038f-41c4-934b-84666af325f3\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") "
Jul 11 04:24:47.291292 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.291049    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fc479e-038f-41c4-934b-84666af325f3-tls-certs\") pod \"f9fc479e-038f-41c4-934b-84666af325f3\" (UID: \"f9fc479e-038f-41c4-934b-84666af325f3\") "
Jul 11 04:24:47.291292 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.291222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-home" (OuterVolumeSpecName: "home") pod "f9fc479e-038f-41c4-934b-84666af325f3" (UID: "f9fc479e-038f-41c4-934b-84666af325f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:47.291462 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.291339    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9fc479e-038f-41c4-934b-84666af325f3" (UID: "f9fc479e-038f-41c4-934b-84666af325f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:47.291537 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.291483    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:47.291537 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.291502    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:47.291637 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.291558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9fc479e-038f-41c4-934b-84666af325f3" (UID: "f9fc479e-038f-41c4-934b-84666af325f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:47.294055 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.294024    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fc479e-038f-41c4-934b-84666af325f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9fc479e-038f-41c4-934b-84666af325f3" (UID: "f9fc479e-038f-41c4-934b-84666af325f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:24:47.294168 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.294130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-dshm" (OuterVolumeSpecName: "dshm") pod "f9fc479e-038f-41c4-934b-84666af325f3" (UID: "f9fc479e-038f-41c4-934b-84666af325f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:47.294212 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.294162    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fc479e-038f-41c4-934b-84666af325f3-kube-api-access-h2jcs" (OuterVolumeSpecName: "kube-api-access-h2jcs") pod "f9fc479e-038f-41c4-934b-84666af325f3" (UID: "f9fc479e-038f-41c4-934b-84666af325f3"). InnerVolumeSpecName "kube-api-access-h2jcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:24:47.392632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.392596    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:47.392632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.392626    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fc479e-038f-41c4-934b-84666af325f3-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:47.392632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.392637    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jcs\" (UniqueName: \"kubernetes.io/projected/f9fc479e-038f-41c4-934b-84666af325f3-kube-api-access-h2jcs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:47.392874 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.392646    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fc479e-038f-41c4-934b-84666af325f3-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:47.707935 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:47.707899    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:24:48.005563 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.005529    2582 generic.go:356] "Generic (PLEG): container finished" podID="f9fc479e-038f-41c4-934b-84666af325f3" containerID="74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab" exitCode=0
Jul 11 04:24:48.005755 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.005581    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"
Jul 11 04:24:48.005755 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.005625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6" event={"ID":"f9fc479e-038f-41c4-934b-84666af325f3","Type":"ContainerDied","Data":"74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab"}
Jul 11 04:24:48.005755 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.005675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6" event={"ID":"f9fc479e-038f-41c4-934b-84666af325f3","Type":"ContainerDied","Data":"4e86742df388db0df131dbb33649af0a75bdd5fc5e7dab5a9f28e9d1305015b4"}
Jul 11 04:24:48.005755 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.005698    2582 scope.go:117] "RemoveContainer" containerID="74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab"
Jul 11 04:24:48.006223 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.006068    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" podUID="7884daa9-4fb8-498f-9602-9db9c5e485a7" containerName="main" containerID="cri-o://ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec" gracePeriod=45
Jul 11 04:24:48.015029 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.015012    2582 scope.go:117] "RemoveContainer" containerID="74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab"
Jul 11 04:24:48.015313 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:24:48.015295    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab\": container with ID starting with 74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab not found: ID does not exist" containerID="74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab"
Jul 11 04:24:48.015382 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.015322    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab"} err="failed to get container status \"74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab\": rpc error: code = NotFound desc = could not find container \"74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab\": container with ID starting with 74fd978385fbcc7fa4212ed6ba8b1c9acc8d3de459526e8732290a97eeb034ab not found: ID does not exist"
Jul 11 04:24:48.021703 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.021682    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"]
Jul 11 04:24:48.023551 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.023532    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6ee78d/auth-test-model-6ee78d-kserve-5d956976c-82jn6"]
Jul 11 04:24:48.271816 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.271755    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:24:48.302297 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/7884daa9-4fb8-498f-9602-9db9c5e485a7-kube-api-access-drf8t\") pod \"7884daa9-4fb8-498f-9602-9db9c5e485a7\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") "
Jul 11 04:24:48.302440 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-tmp-dir\") pod \"7884daa9-4fb8-498f-9602-9db9c5e485a7\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") "
Jul 11 04:24:48.302440 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302354    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-dshm\") pod \"7884daa9-4fb8-498f-9602-9db9c5e485a7\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") "
Jul 11 04:24:48.302547 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302531    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-model-cache\") pod \"7884daa9-4fb8-498f-9602-9db9c5e485a7\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") "
Jul 11 04:24:48.302616 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302600    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-home\") pod \"7884daa9-4fb8-498f-9602-9db9c5e485a7\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") "
Jul 11 04:24:48.302683 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302623    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7884daa9-4fb8-498f-9602-9db9c5e485a7" (UID: "7884daa9-4fb8-498f-9602-9db9c5e485a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:48.302683 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302653    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7884daa9-4fb8-498f-9602-9db9c5e485a7-tls-certs\") pod \"7884daa9-4fb8-498f-9602-9db9c5e485a7\" (UID: \"7884daa9-4fb8-498f-9602-9db9c5e485a7\") "
Jul 11 04:24:48.302802 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302779    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7884daa9-4fb8-498f-9602-9db9c5e485a7" (UID: "7884daa9-4fb8-498f-9602-9db9c5e485a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:48.302802 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302790    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-home" (OuterVolumeSpecName: "home") pod "7884daa9-4fb8-498f-9602-9db9c5e485a7" (UID: "7884daa9-4fb8-498f-9602-9db9c5e485a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:48.302977 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302951    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:48.303294 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302984    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:48.303294 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.302998    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:48.304807 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.304780    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7884daa9-4fb8-498f-9602-9db9c5e485a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7884daa9-4fb8-498f-9602-9db9c5e485a7" (UID: "7884daa9-4fb8-498f-9602-9db9c5e485a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:24:48.304807 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.304792    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-dshm" (OuterVolumeSpecName: "dshm") pod "7884daa9-4fb8-498f-9602-9db9c5e485a7" (UID: "7884daa9-4fb8-498f-9602-9db9c5e485a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:24:48.304934 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.304837    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7884daa9-4fb8-498f-9602-9db9c5e485a7-kube-api-access-drf8t" (OuterVolumeSpecName: "kube-api-access-drf8t") pod "7884daa9-4fb8-498f-9602-9db9c5e485a7" (UID: "7884daa9-4fb8-498f-9602-9db9c5e485a7"). InnerVolumeSpecName "kube-api-access-drf8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:24:48.403618 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.403583    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/7884daa9-4fb8-498f-9602-9db9c5e485a7-kube-api-access-drf8t\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:48.403618 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.403613    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7884daa9-4fb8-498f-9602-9db9c5e485a7-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:48.403618 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:48.403623    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7884daa9-4fb8-498f-9602-9db9c5e485a7-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:24:49.015019 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.014976    2582 generic.go:356] "Generic (PLEG): container finished" podID="7884daa9-4fb8-498f-9602-9db9c5e485a7" containerID="ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec" exitCode=0
Jul 11 04:24:49.015534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.015030    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"
Jul 11 04:24:49.015534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.015046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" event={"ID":"7884daa9-4fb8-498f-9602-9db9c5e485a7","Type":"ContainerDied","Data":"ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec"}
Jul 11 04:24:49.015534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.015089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" event={"ID":"7884daa9-4fb8-498f-9602-9db9c5e485a7","Type":"ContainerDied","Data":"ad64698cdc9aed8a89c8e36ab14f48b90a625becf328be721418fbc153be35c4"}
Jul 11 04:24:49.015534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.015122    2582 scope.go:117] "RemoveContainer" containerID="ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec"
Jul 11 04:24:49.024753 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.024735    2582 scope.go:117] "RemoveContainer" containerID="ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec"
Jul 11 04:24:49.025007 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:24:49.024991    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec\": container with ID starting with ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec not found: ID does not exist" containerID="ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec"
Jul 11 04:24:49.025054 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.025018    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec"} err="failed to get container status \"ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec\": rpc error: code = NotFound desc = could not find container \"ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec\": container with ID starting with ff36ea3e5ca551ab6e4e233661edc7b4a5370ea30489bfdfeb487840b361d5ec not found: ID does not exist"
Jul 11 04:24:49.035417 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.035389    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"]
Jul 11 04:24:49.039034 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.039014    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx"]
Jul 11 04:24:49.645748 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.645712    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7884daa9-4fb8-498f-9602-9db9c5e485a7" path="/var/lib/kubelet/pods/7884daa9-4fb8-498f-9602-9db9c5e485a7/volumes"
Jul 11 04:24:49.646133 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:49.646091    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fc479e-038f-41c4-934b-84666af325f3" path="/var/lib/kubelet/pods/f9fc479e-038f-41c4-934b-84666af325f3/volumes"
Jul 11 04:24:53.246073 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:53.246022    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-1da175/auth-test-model-1da175-kserve-57677f8b87-zc7kx" podUID="7884daa9-4fb8-498f-9602-9db9c5e485a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Jul 11 04:24:59.911456 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.911421    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"]
Jul 11 04:24:59.912203 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.912183    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7884daa9-4fb8-498f-9602-9db9c5e485a7" containerName="main"
Jul 11 04:24:59.912203 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.912204    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7884daa9-4fb8-498f-9602-9db9c5e485a7" containerName="main"
Jul 11 04:24:59.912359 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.912229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fc479e-038f-41c4-934b-84666af325f3" containerName="main"
Jul 11 04:24:59.912359 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.912239    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fc479e-038f-41c4-934b-84666af325f3" containerName="main"
Jul 11 04:24:59.912359 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.912332    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fc479e-038f-41c4-934b-84666af325f3" containerName="main"
Jul 11 04:24:59.912359 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.912346    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7884daa9-4fb8-498f-9602-9db9c5e485a7" containerName="main"
Jul 11 04:24:59.916896 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.916875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:24:59.920049 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.920028    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"default-dockercfg-5slww\""
Jul 11 04:24:59.920049 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.920047    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"sub-test-model-1da175-kserve-self-signed-certs\""
Jul 11 04:24:59.920049 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.920030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"kube-root-ca.crt\""
Jul 11 04:24:59.920333 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.920029    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"openshift-service-ca.crt\""
Jul 11 04:24:59.924445 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.924420    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"]
Jul 11 04:24:59.938879 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:24:59.938856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:00.009976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.009937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5pb\" (UniqueName: \"kubernetes.io/projected/33d96183-0240-4016-a69c-b84c44013975-kube-api-access-md5pb\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.009976 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.009979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-model-cache\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.010267 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.010051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-dshm\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.010267 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.010083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-tmp-dir\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.010267 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.010175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-home\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.010267 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.010205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d96183-0240-4016-a69c-b84c44013975-tls-certs\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.110649 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.110615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d96183-0240-4016-a69c-b84c44013975-tls-certs\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.110862 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.110654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5pb\" (UniqueName: \"kubernetes.io/projected/33d96183-0240-4016-a69c-b84c44013975-kube-api-access-md5pb\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.110862 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.110676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-model-cache\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.110862 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.110740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-dshm\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.110862 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.110779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-tmp-dir\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.111085 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.110915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-home\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.111222 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.111200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-model-cache\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.111262 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.111245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-tmp-dir\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.111297 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.111287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-home\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.113163 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.113144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-dshm\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.113491 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.113469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d96183-0240-4016-a69c-b84c44013975-tls-certs\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.118559 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.118533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5pb\" (UniqueName: \"kubernetes.io/projected/33d96183-0240-4016-a69c-b84c44013975-kube-api-access-md5pb\") pod \"sub-test-model-1da175-kserve-5d6bf65c89-tzmd2\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") " pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.227350 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.227259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:00.372455 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:00.372394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"]
Jul 11 04:25:00.375009 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:25:00.374974    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33d96183_0240_4016_a69c_b84c44013975.slice/crio-0b749fe3558a1a0c97bbb25b0f4dc18e78ccfa6f51ba9559df80ad1a41574f30 WatchSource:0}: Error finding container 0b749fe3558a1a0c97bbb25b0f4dc18e78ccfa6f51ba9559df80ad1a41574f30: Status 404 returned error can't find the container with id 0b749fe3558a1a0c97bbb25b0f4dc18e78ccfa6f51ba9559df80ad1a41574f30
Jul 11 04:25:01.079708 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:01.079672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2" event={"ID":"33d96183-0240-4016-a69c-b84c44013975","Type":"ContainerStarted","Data":"c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f"}
Jul 11 04:25:01.079708 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:01.079710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2" event={"ID":"33d96183-0240-4016-a69c-b84c44013975","Type":"ContainerStarted","Data":"0b749fe3558a1a0c97bbb25b0f4dc18e78ccfa6f51ba9559df80ad1a41574f30"}
Jul 11 04:25:01.095850 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:01.095803    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2" podStartSLOduration=2.09578964 podStartE2EDuration="2.09578964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:25:01.094127687 +0000 UTC m=+2545.963574245" watchObservedRunningTime="2026-07-11 04:25:01.09578964 +0000 UTC m=+2545.965236205"
Jul 11 04:25:06.814314 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:06.814278    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:25:08.407416 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:08.407377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:10.227801 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:10.227762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:10.228250 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:10.227815    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:10.240245 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:10.240222    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:11.140602 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:11.140571    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:25:28.133851 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:28.133810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:28.187288 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:28.187254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:30.415307 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:30.415266    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:25:31.133215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:31.133175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:31.237114 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:31.237076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:39.232109 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:39.232064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:39.333268 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:39.333226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:45.711642 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:45.711591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:46.629806 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:46.629774    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"]
Jul 11 04:25:46.646738 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:46.646708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:25:50.901906 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:25:50.901870    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:26:01.302629 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.302524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:26:01.650583 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.650469    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2" podUID="33d96183-0240-4016-a69c-b84c44013975" containerName="main" containerID="cri-o://c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f" gracePeriod=45
Jul 11 04:26:01.912778 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.912714    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:26:01.985259 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985224    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-home\") pod \"33d96183-0240-4016-a69c-b84c44013975\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") "
Jul 11 04:26:01.985443 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-model-cache\") pod \"33d96183-0240-4016-a69c-b84c44013975\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") "
Jul 11 04:26:01.985443 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985296    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d96183-0240-4016-a69c-b84c44013975-tls-certs\") pod \"33d96183-0240-4016-a69c-b84c44013975\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") "
Jul 11 04:26:01.985443 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-dshm\") pod \"33d96183-0240-4016-a69c-b84c44013975\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") "
Jul 11 04:26:01.985443 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-tmp-dir\") pod \"33d96183-0240-4016-a69c-b84c44013975\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") "
Jul 11 04:26:01.985660 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985454    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md5pb\" (UniqueName: \"kubernetes.io/projected/33d96183-0240-4016-a69c-b84c44013975-kube-api-access-md5pb\") pod \"33d96183-0240-4016-a69c-b84c44013975\" (UID: \"33d96183-0240-4016-a69c-b84c44013975\") "
Jul 11 04:26:01.985660 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985518    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-home" (OuterVolumeSpecName: "home") pod "33d96183-0240-4016-a69c-b84c44013975" (UID: "33d96183-0240-4016-a69c-b84c44013975"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:26:01.985660 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985539    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-model-cache" (OuterVolumeSpecName: "model-cache") pod "33d96183-0240-4016-a69c-b84c44013975" (UID: "33d96183-0240-4016-a69c-b84c44013975"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:26:01.985660 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985647    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33d96183-0240-4016-a69c-b84c44013975" (UID: "33d96183-0240-4016-a69c-b84c44013975"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:26:01.985859 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985787    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:26:01.985859 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985802    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:26:01.985859 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.985817    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:26:01.987598 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.987572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-dshm" (OuterVolumeSpecName: "dshm") pod "33d96183-0240-4016-a69c-b84c44013975" (UID: "33d96183-0240-4016-a69c-b84c44013975"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:26:01.987598 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.987573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d96183-0240-4016-a69c-b84c44013975-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33d96183-0240-4016-a69c-b84c44013975" (UID: "33d96183-0240-4016-a69c-b84c44013975"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:26:01.987775 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:01.987690    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d96183-0240-4016-a69c-b84c44013975-kube-api-access-md5pb" (OuterVolumeSpecName: "kube-api-access-md5pb") pod "33d96183-0240-4016-a69c-b84c44013975" (UID: "33d96183-0240-4016-a69c-b84c44013975"). InnerVolumeSpecName "kube-api-access-md5pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:26:02.087400 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.087352    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md5pb\" (UniqueName: \"kubernetes.io/projected/33d96183-0240-4016-a69c-b84c44013975-kube-api-access-md5pb\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:26:02.087400 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.087393    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d96183-0240-4016-a69c-b84c44013975-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:26:02.087400 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.087407    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d96183-0240-4016-a69c-b84c44013975-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:26:02.402700 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.402662    2582 generic.go:356] "Generic (PLEG): container finished" podID="33d96183-0240-4016-a69c-b84c44013975" containerID="c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f" exitCode=0
Jul 11 04:26:02.403130 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.402714    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"
Jul 11 04:26:02.403130 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.402753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2" event={"ID":"33d96183-0240-4016-a69c-b84c44013975","Type":"ContainerDied","Data":"c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f"}
Jul 11 04:26:02.403130 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.402798    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2" event={"ID":"33d96183-0240-4016-a69c-b84c44013975","Type":"ContainerDied","Data":"0b749fe3558a1a0c97bbb25b0f4dc18e78ccfa6f51ba9559df80ad1a41574f30"}
Jul 11 04:26:02.403130 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.402821    2582 scope.go:117] "RemoveContainer" containerID="c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f"
Jul 11 04:26:02.412437 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.412421    2582 scope.go:117] "RemoveContainer" containerID="c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f"
Jul 11 04:26:02.412695 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:26:02.412677    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f\": container with ID starting with c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f not found: ID does not exist" containerID="c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f"
Jul 11 04:26:02.412744 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.412705    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f"} err="failed to get container status \"c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f\": rpc error: code = NotFound desc = could not find container \"c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f\": container with ID starting with c6c5a2f66e4cd7021010816d497c9742a4272c66603382ae188ce91d2fc4754f not found: ID does not exist"
Jul 11 04:26:02.423468 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.423424    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"]
Jul 11 04:26:02.425405 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:02.425385    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/sub-test-model-1da175-kserve-5d6bf65c89-tzmd2"]
Jul 11 04:26:03.646849 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:03.646811    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d96183-0240-4016-a69c-b84c44013975" path="/var/lib/kubelet/pods/33d96183-0240-4016-a69c-b84c44013975/volumes"
Jul 11 04:26:07.109806 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:07.109770    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:26:29.491618 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.491578    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"]
Jul 11 04:26:29.492179 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.492153    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d96183-0240-4016-a69c-b84c44013975" containerName="main"
Jul 11 04:26:29.492179 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.492179    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d96183-0240-4016-a69c-b84c44013975" containerName="main"
Jul 11 04:26:29.492411 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.492290    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d96183-0240-4016-a69c-b84c44013975" containerName="main"
Jul 11 04:26:29.494749 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.494731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.497015 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.496988    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"kube-root-ca.crt\""
Jul 11 04:26:29.497201 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.497179    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"openshift-service-ca.crt\""
Jul 11 04:26:29.498079 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.498060    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"default-dockercfg-5slww\""
Jul 11 04:26:29.498206 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.498130    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1da175\"/\"rate-test-model-1da175-kserve-self-signed-certs\""
Jul 11 04:26:29.504786 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.504763    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"]
Jul 11 04:26:29.513449 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.513416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:26:29.546290 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.546254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-home\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.546465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.546303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-model-cache\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.546465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.546383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-tmp-dir\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.546465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.546407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-dshm\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.546465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.546434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247ef854-9c00-4117-aaa4-c3607f5488e8-tls-certs\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.546465 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.546457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5wt\" (UniqueName: \"kubernetes.io/projected/247ef854-9c00-4117-aaa4-c3607f5488e8-kube-api-access-8x5wt\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647426 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-tmp-dir\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-dshm\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247ef854-9c00-4117-aaa4-c3607f5488e8-tls-certs\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5wt\" (UniqueName: \"kubernetes.io/projected/247ef854-9c00-4117-aaa4-c3607f5488e8-kube-api-access-8x5wt\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-home\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647632 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-model-cache\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-tmp-dir\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-model-cache\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.647990 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.647919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-home\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.649872 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.649851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-dshm\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.650037 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.650018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247ef854-9c00-4117-aaa4-c3607f5488e8-tls-certs\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.655167 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.655138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5wt\" (UniqueName: \"kubernetes.io/projected/247ef854-9c00-4117-aaa4-c3607f5488e8-kube-api-access-8x5wt\") pod \"rate-test-model-1da175-kserve-7ff65dd899-vjnzs\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") " pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.804888 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.804792    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:29.944916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:29.944883    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"]
Jul 11 04:26:29.947760 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:26:29.947730    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod247ef854_9c00_4117_aaa4_c3607f5488e8.slice/crio-e74be4547a018f2534d637d768a5e919a1ebbab7c0ab74afa06eb05db6ac9871 WatchSource:0}: Error finding container e74be4547a018f2534d637d768a5e919a1ebbab7c0ab74afa06eb05db6ac9871: Status 404 returned error can't find the container with id e74be4547a018f2534d637d768a5e919a1ebbab7c0ab74afa06eb05db6ac9871
Jul 11 04:26:30.547358 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:30.547310    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs" event={"ID":"247ef854-9c00-4117-aaa4-c3607f5488e8","Type":"ContainerStarted","Data":"67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059"}
Jul 11 04:26:30.547358 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:30.547367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs" event={"ID":"247ef854-9c00-4117-aaa4-c3607f5488e8","Type":"ContainerStarted","Data":"e74be4547a018f2534d637d768a5e919a1ebbab7c0ab74afa06eb05db6ac9871"}
Jul 11 04:26:30.564150 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:30.564075    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs" podStartSLOduration=1.564058947 podStartE2EDuration="1.564058947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:26:30.562938192 +0000 UTC m=+2635.432384769" watchObservedRunningTime="2026-07-11 04:26:30.564058947 +0000 UTC m=+2635.433505564"
Jul 11 04:26:34.102765 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:34.102727    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:26:39.805037 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:39.804984    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:39.805452 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:39.805056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:39.817346 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:39.817323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:40.610912 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:40.610880    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:26:48.089062 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:48.089021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:26:48.188344 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:48.188281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:26:52.001426 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:52.001391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:26:56.311687 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:56.311652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:26:56.410223 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:26:56.410188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:01.292403 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:01.292364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:07.744552 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:07.744516    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:27:14.946701 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:14.946664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:15.814198 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:15.814164    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"]
Jul 11 04:27:15.831922 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:15.831892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:22.310568 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:22.310534    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:27:30.835577 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:30.835436    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs" podUID="247ef854-9c00-4117-aaa4-c3607f5488e8" containerName="main" containerID="cri-o://67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059" gracePeriod=45
Jul 11 04:27:31.103325 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.103301    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:27:31.226046 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226007    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-tmp-dir\") pod \"247ef854-9c00-4117-aaa4-c3607f5488e8\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") "
Jul 11 04:27:31.226046 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226048    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-home\") pod \"247ef854-9c00-4117-aaa4-c3607f5488e8\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") "
Jul 11 04:27:31.226321 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247ef854-9c00-4117-aaa4-c3607f5488e8-tls-certs\") pod \"247ef854-9c00-4117-aaa4-c3607f5488e8\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") "
Jul 11 04:27:31.226321 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-model-cache\") pod \"247ef854-9c00-4117-aaa4-c3607f5488e8\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") "
Jul 11 04:27:31.226321 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-dshm\") pod \"247ef854-9c00-4117-aaa4-c3607f5488e8\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") "
Jul 11 04:27:31.226321 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x5wt\" (UniqueName: \"kubernetes.io/projected/247ef854-9c00-4117-aaa4-c3607f5488e8-kube-api-access-8x5wt\") pod \"247ef854-9c00-4117-aaa4-c3607f5488e8\" (UID: \"247ef854-9c00-4117-aaa4-c3607f5488e8\") "
Jul 11 04:27:31.226534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226375    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-home" (OuterVolumeSpecName: "home") pod "247ef854-9c00-4117-aaa4-c3607f5488e8" (UID: "247ef854-9c00-4117-aaa4-c3607f5488e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:27:31.226534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "247ef854-9c00-4117-aaa4-c3607f5488e8" (UID: "247ef854-9c00-4117-aaa4-c3607f5488e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:27:31.226534 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226474    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "247ef854-9c00-4117-aaa4-c3607f5488e8" (UID: "247ef854-9c00-4117-aaa4-c3607f5488e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:27:31.226669 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226567    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-tmp-dir\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:31.226669 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226587    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-home\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:31.226669 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.226601    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-model-cache\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:31.228469 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.228447    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247ef854-9c00-4117-aaa4-c3607f5488e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "247ef854-9c00-4117-aaa4-c3607f5488e8" (UID: "247ef854-9c00-4117-aaa4-c3607f5488e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:27:31.228588 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.228557    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-dshm" (OuterVolumeSpecName: "dshm") pod "247ef854-9c00-4117-aaa4-c3607f5488e8" (UID: "247ef854-9c00-4117-aaa4-c3607f5488e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 11 04:27:31.228588 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.228562    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247ef854-9c00-4117-aaa4-c3607f5488e8-kube-api-access-8x5wt" (OuterVolumeSpecName: "kube-api-access-8x5wt") pod "247ef854-9c00-4117-aaa4-c3607f5488e8" (UID: "247ef854-9c00-4117-aaa4-c3607f5488e8"). InnerVolumeSpecName "kube-api-access-8x5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:27:31.327410 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.327376    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247ef854-9c00-4117-aaa4-c3607f5488e8-tls-certs\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:31.327410 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.327405    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247ef854-9c00-4117-aaa4-c3607f5488e8-dshm\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:31.327410 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.327415    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x5wt\" (UniqueName: \"kubernetes.io/projected/247ef854-9c00-4117-aaa4-c3607f5488e8-kube-api-access-8x5wt\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:31.861319 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.861285    2582 generic.go:356] "Generic (PLEG): container finished" podID="247ef854-9c00-4117-aaa4-c3607f5488e8" containerID="67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059" exitCode=0
Jul 11 04:27:31.861777 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.861344    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"
Jul 11 04:27:31.861777 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.861376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs" event={"ID":"247ef854-9c00-4117-aaa4-c3607f5488e8","Type":"ContainerDied","Data":"67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059"}
Jul 11 04:27:31.861777 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.861418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs" event={"ID":"247ef854-9c00-4117-aaa4-c3607f5488e8","Type":"ContainerDied","Data":"e74be4547a018f2534d637d768a5e919a1ebbab7c0ab74afa06eb05db6ac9871"}
Jul 11 04:27:31.861777 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.861437    2582 scope.go:117] "RemoveContainer" containerID="67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059"
Jul 11 04:27:31.872325 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.872306    2582 scope.go:117] "RemoveContainer" containerID="67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059"
Jul 11 04:27:31.872594 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:27:31.872566    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059\": container with ID starting with 67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059 not found: ID does not exist" containerID="67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059"
Jul 11 04:27:31.872649 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.872596    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059"} err="failed to get container status \"67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059\": rpc error: code = NotFound desc = could not find container \"67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059\": container with ID starting with 67d31b0b5ad017513a7229d9c572c232f6ae3326c21cad231dc0fd2fa9dcb059 not found: ID does not exist"
Jul 11 04:27:31.880723 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.880693    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"]
Jul 11 04:27:31.884266 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:31.884240    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1da175/rate-test-model-1da175-kserve-7ff65dd899-vjnzs"]
Jul 11 04:27:33.646613 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:33.646581    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247ef854-9c00-4117-aaa4-c3607f5488e8" path="/var/lib/kubelet/pods/247ef854-9c00-4117-aaa4-c3607f5488e8/volumes"
Jul 11 04:27:35.090877 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:35.090842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:35.151384 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:35.151347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:35.252175 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:35.252143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:35.927061 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:35.927031    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:27:35.936228 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:35.936206    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:27:45.006665 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.006628    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"]
Jul 11 04:27:45.007227 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.006885    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44" podUID="2446db14-eb6e-4176-b3d2-344090fe9fee" containerName="maas-api" containerID="cri-o://ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad" gracePeriod=30
Jul 11 04:27:45.031200 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.031174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:45.271433 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.271363    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:27:45.349409 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.349364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2446db14-eb6e-4176-b3d2-344090fe9fee-maas-api-tls\") pod \"2446db14-eb6e-4176-b3d2-344090fe9fee\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") "
Jul 11 04:27:45.349593 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.349432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2446db14-eb6e-4176-b3d2-344090fe9fee-openshift-service-ca\") pod \"2446db14-eb6e-4176-b3d2-344090fe9fee\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") "
Jul 11 04:27:45.349593 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.349475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drq5v\" (UniqueName: \"kubernetes.io/projected/2446db14-eb6e-4176-b3d2-344090fe9fee-kube-api-access-drq5v\") pod \"2446db14-eb6e-4176-b3d2-344090fe9fee\" (UID: \"2446db14-eb6e-4176-b3d2-344090fe9fee\") "
Jul 11 04:27:45.349824 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.349797    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2446db14-eb6e-4176-b3d2-344090fe9fee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2446db14-eb6e-4176-b3d2-344090fe9fee" (UID: "2446db14-eb6e-4176-b3d2-344090fe9fee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:27:45.351761 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.351723    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2446db14-eb6e-4176-b3d2-344090fe9fee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2446db14-eb6e-4176-b3d2-344090fe9fee" (UID: "2446db14-eb6e-4176-b3d2-344090fe9fee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:27:45.352073 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.352040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2446db14-eb6e-4176-b3d2-344090fe9fee-kube-api-access-drq5v" (OuterVolumeSpecName: "kube-api-access-drq5v") pod "2446db14-eb6e-4176-b3d2-344090fe9fee" (UID: "2446db14-eb6e-4176-b3d2-344090fe9fee"). InnerVolumeSpecName "kube-api-access-drq5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:27:45.450235 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.450174    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2446db14-eb6e-4176-b3d2-344090fe9fee-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:45.450235 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.450230    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drq5v\" (UniqueName: \"kubernetes.io/projected/2446db14-eb6e-4176-b3d2-344090fe9fee-kube-api-access-drq5v\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:45.450235 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.450247    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2446db14-eb6e-4176-b3d2-344090fe9fee-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:27:45.935599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.935491    2582 generic.go:356] "Generic (PLEG): container finished" podID="2446db14-eb6e-4176-b3d2-344090fe9fee" containerID="ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad" exitCode=0
Jul 11 04:27:45.935599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.935544    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"
Jul 11 04:27:45.935599 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.935576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44" event={"ID":"2446db14-eb6e-4176-b3d2-344090fe9fee","Type":"ContainerDied","Data":"ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad"}
Jul 11 04:27:45.935891 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.935642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44" event={"ID":"2446db14-eb6e-4176-b3d2-344090fe9fee","Type":"ContainerDied","Data":"317b9426dd135834fccc902dd55ce610e0c15f50a8ecb61453a5b0efbbad1183"}
Jul 11 04:27:45.935891 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.935671    2582 scope.go:117] "RemoveContainer" containerID="ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad"
Jul 11 04:27:45.945427 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.945407    2582 scope.go:117] "RemoveContainer" containerID="ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad"
Jul 11 04:27:45.945728 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:27:45.945709    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad\": container with ID starting with ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad not found: ID does not exist" containerID="ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad"
Jul 11 04:27:45.945800 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.945737    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad"} err="failed to get container status \"ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad\": rpc error: code = NotFound desc = could not find container \"ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad\": container with ID starting with ed2ed222553da5029e50437c298185d728a87dd52ae5805f086fc664953875ad not found: ID does not exist"
Jul 11 04:27:45.954162 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.954090    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"]
Jul 11 04:27:45.958272 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:45.958248    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6ee78d-678f869cf-9hz44"]
Jul 11 04:27:47.646699 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:47.646666    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2446db14-eb6e-4176-b3d2-344090fe9fee" path="/var/lib/kubelet/pods/2446db14-eb6e-4176-b3d2-344090fe9fee/volumes"
Jul 11 04:27:48.600052 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:48.600014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:27:49.714355 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:49.714311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:55.643887 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:55.643854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:27:55.741118 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:27:55.741079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:00.416143 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.416111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:00.701266 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.701176    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:00.777811 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.777767    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"]
Jul 11 04:28:00.778223 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.778206    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2446db14-eb6e-4176-b3d2-344090fe9fee" containerName="maas-api"
Jul 11 04:28:00.778223 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.778224    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2446db14-eb6e-4176-b3d2-344090fe9fee" containerName="maas-api"
Jul 11 04:28:00.778373 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.778246    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247ef854-9c00-4117-aaa4-c3607f5488e8" containerName="main"
Jul 11 04:28:00.778373 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.778254    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="247ef854-9c00-4117-aaa4-c3607f5488e8" containerName="main"
Jul 11 04:28:00.778373 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.778342    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2446db14-eb6e-4176-b3d2-344090fe9fee" containerName="maas-api"
Jul 11 04:28:00.778373 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.778359    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="247ef854-9c00-4117-aaa4-c3607f5488e8" containerName="main"
Jul 11 04:28:00.782963 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.782942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:00.785574 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.785549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\""
Jul 11 04:28:00.787513 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.787489    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"]
Jul 11 04:28:00.892051 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.892011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a73028c-6ffa-437d-a782-f9e69fc88d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4\" (UID: \"2a73028c-6ffa-437d-a782-f9e69fc88d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:00.992533 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.992507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a73028c-6ffa-437d-a782-f9e69fc88d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4\" (UID: \"2a73028c-6ffa-437d-a782-f9e69fc88d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:00.993190 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:00.993171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a73028c-6ffa-437d-a782-f9e69fc88d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4\" (UID: \"2a73028c-6ffa-437d-a782-f9e69fc88d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:01.096579 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:01.096542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:01.249338 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:01.249311    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"]
Jul 11 04:28:01.251374 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:28:01.251346    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a73028c_6ffa_437d_a782_f9e69fc88d2b.slice/crio-0d33ce469463bd643a2f1d01755fc052f64c1267286fcca1dba8ef99db258dac WatchSource:0}: Error finding container 0d33ce469463bd643a2f1d01755fc052f64c1267286fcca1dba8ef99db258dac: Status 404 returned error can't find the container with id 0d33ce469463bd643a2f1d01755fc052f64c1267286fcca1dba8ef99db258dac
Jul 11 04:28:01.366528 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:01.366470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:01.466890 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:01.466848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:02.020665 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:02.020627    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a73028c-6ffa-437d-a782-f9e69fc88d2b" containerID="5c6b4830eabbb77c4e896851283ba091f7fb41c496abc798ca5a88d86466e85d" exitCode=0
Jul 11 04:28:02.020873 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:02.020804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4" event={"ID":"2a73028c-6ffa-437d-a782-f9e69fc88d2b","Type":"ContainerDied","Data":"5c6b4830eabbb77c4e896851283ba091f7fb41c496abc798ca5a88d86466e85d"}
Jul 11 04:28:02.020945 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:02.020896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4" event={"ID":"2a73028c-6ffa-437d-a782-f9e69fc88d2b","Type":"ContainerStarted","Data":"0d33ce469463bd643a2f1d01755fc052f64c1267286fcca1dba8ef99db258dac"}
Jul 11 04:28:03.164451 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:03.164424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:03.213702 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:03.213662    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a73028c-6ffa-437d-a782-f9e69fc88d2b-maas-api-service-ca\") pod \"2a73028c-6ffa-437d-a782-f9e69fc88d2b\" (UID: \"2a73028c-6ffa-437d-a782-f9e69fc88d2b\") "
Jul 11 04:28:03.214025 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:03.213999    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a73028c-6ffa-437d-a782-f9e69fc88d2b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a73028c-6ffa-437d-a782-f9e69fc88d2b" (UID: "2a73028c-6ffa-437d-a782-f9e69fc88d2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:28:03.314349 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:03.314254    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a73028c-6ffa-437d-a782-f9e69fc88d2b-maas-api-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:28:04.032705 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:04.032676    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4"
Jul 11 04:28:04.032916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:04.032722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4" event={"ID":"2a73028c-6ffa-437d-a782-f9e69fc88d2b","Type":"ContainerDied","Data":"0d33ce469463bd643a2f1d01755fc052f64c1267286fcca1dba8ef99db258dac"}
Jul 11 04:28:04.032916 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:04.032768    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d33ce469463bd643a2f1d01755fc052f64c1267286fcca1dba8ef99db258dac"
Jul 11 04:28:11.151678 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.151647    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"]
Jul 11 04:28:11.152158 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.151870    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" podUID="4f744fe7-7e56-41c8-8704-0b829ddb2044" containerName="maas-api" containerID="cri-o://f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b" gracePeriod=30
Jul 11 04:28:11.178266 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.178230    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:11.615311 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.615284    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:28:11.698545 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.698442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p77s8\" (UniqueName: \"kubernetes.io/projected/4f744fe7-7e56-41c8-8704-0b829ddb2044-kube-api-access-p77s8\") pod \"4f744fe7-7e56-41c8-8704-0b829ddb2044\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") "
Jul 11 04:28:11.698729 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.698561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls\") pod \"4f744fe7-7e56-41c8-8704-0b829ddb2044\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") "
Jul 11 04:28:11.698729 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.698617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f744fe7-7e56-41c8-8704-0b829ddb2044-openshift-service-ca\") pod \"4f744fe7-7e56-41c8-8704-0b829ddb2044\" (UID: \"4f744fe7-7e56-41c8-8704-0b829ddb2044\") "
Jul 11 04:28:11.698997 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.698968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f744fe7-7e56-41c8-8704-0b829ddb2044-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f744fe7-7e56-41c8-8704-0b829ddb2044" (UID: "4f744fe7-7e56-41c8-8704-0b829ddb2044"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 11 04:28:11.700691 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.700663    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f744fe7-7e56-41c8-8704-0b829ddb2044-kube-api-access-p77s8" (OuterVolumeSpecName: "kube-api-access-p77s8") pod "4f744fe7-7e56-41c8-8704-0b829ddb2044" (UID: "4f744fe7-7e56-41c8-8704-0b829ddb2044"). InnerVolumeSpecName "kube-api-access-p77s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 11 04:28:11.700691 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.700693    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f744fe7-7e56-41c8-8704-0b829ddb2044" (UID: "4f744fe7-7e56-41c8-8704-0b829ddb2044"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 11 04:28:11.800023 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.799984    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f744fe7-7e56-41c8-8704-0b829ddb2044-maas-api-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:28:11.800023 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.800022    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f744fe7-7e56-41c8-8704-0b829ddb2044-openshift-service-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:28:11.800248 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:11.800037    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77s8\" (UniqueName: \"kubernetes.io/projected/4f744fe7-7e56-41c8-8704-0b829ddb2044-kube-api-access-p77s8\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\""
Jul 11 04:28:12.082195 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.082160    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f744fe7-7e56-41c8-8704-0b829ddb2044" containerID="f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b" exitCode=0
Jul 11 04:28:12.082447 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.082203    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"
Jul 11 04:28:12.082447 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.082252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" event={"ID":"4f744fe7-7e56-41c8-8704-0b829ddb2044","Type":"ContainerDied","Data":"f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b"}
Jul 11 04:28:12.082447 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.082310    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj" event={"ID":"4f744fe7-7e56-41c8-8704-0b829ddb2044","Type":"ContainerDied","Data":"773721436dc553fa2074a10d405545ef16abae1b8fb2a3a3b41a501059a2802d"}
Jul 11 04:28:12.082447 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.082339    2582 scope.go:117] "RemoveContainer" containerID="f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b"
Jul 11 04:28:12.092070 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.092053    2582 scope.go:117] "RemoveContainer" containerID="f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b"
Jul 11 04:28:12.092341 ip-10-0-140-230 kubenswrapper[2582]: E0711 04:28:12.092323    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b\": container with ID starting with f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b not found: ID does not exist" containerID="f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b"
Jul 11 04:28:12.092408 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.092353    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b"} err="failed to get container status \"f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b\": rpc error: code = NotFound desc = could not find container \"f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b\": container with ID starting with f6db3cfd02af1ccbf8cb5738e5bd87b70d15e5ff58c125946d671e218c6a9e9b not found: ID does not exist"
Jul 11 04:28:12.103819 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.103796    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"]
Jul 11 04:28:12.107979 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:12.107955    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1da175-8475b7c954-f55jj"]
Jul 11 04:28:13.645672 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:13.645636    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f744fe7-7e56-41c8-8704-0b829ddb2044" path="/var/lib/kubelet/pods/4f744fe7-7e56-41c8-8704-0b829ddb2044/volumes"
Jul 11 04:28:14.206918 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:14.206885    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsnps"]
Jul 11 04:28:15.853669 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:15.853637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:21.606486 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:21.606449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:21.707193 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:21.707163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:26.534011 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:26.533977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:26.775043 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:26.775010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:32.133193 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:32.133166    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bffn4_ff10d8c5-3d3a-4935-a4f6-d193469e6e6c/manager/0.log"
Jul 11 04:28:32.257209 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:32.257182    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vd8l9_9a83a035-1fa4-4f07-9bf4-31be8f6e5661/manager/0.log"
Jul 11 04:28:32.379188 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:32.379070    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57c8d895cd-lmmpf_e03d8538-d22e-4b32-b4af-879a2d0b3704/manager/0.log"
Jul 11 04:28:32.633222 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:32.633192    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4hfnt_b29da052-ad41-4ff0-840e-9eccef268909/manager/0.log"
Jul 11 04:28:32.746383 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:32.746360    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-b8mwq_c7bd2c6b-7412-436a-a48c-a37b239480b8/copy-manifests/0.log"
Jul 11 04:28:32.752713 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:32.752689    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-b8mwq_c7bd2c6b-7412-436a-a48c-a37b239480b8/manager/0.log"
Jul 11 04:28:33.369519 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:33.369491    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c6b47654b-8vfr6_cc66443d-cca6-4b5b-973b-0b31187a5ef7/maas-api/0.log"
Jul 11 04:28:33.491879 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:33.491844    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29729040-qtkd4_8afa7bcd-7cd2-40a3-8960-c1b9b8c5756f/cleanup/0.log"
Jul 11 04:28:34.060178 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:34.060146    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-e88abb-8n68t_48f19ffa-cf06-46de-bcb9-0259af760f30/revoke-keys/0.log"
Jul 11 04:28:34.394847 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:34.394767    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-075325-kzhvk_6be8eab0-ccf2-401a-83df-10f74f2ddc42/revoke-keys/0.log"
Jul 11 04:28:34.935576 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:34.935541    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-1da175-gbkz4_2a73028c-6ffa-437d-a782-f9e69fc88d2b/revoke-keys/0.log"
Jul 11 04:28:36.442488 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:36.442454    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-79jz4_650de894-ccd5-4088-843e-0fbb21075709/manager/0.log"
Jul 11 04:28:36.548831 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:36.548809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kswfr_315db466-92b7-423c-99ef-a6f2c0459199/manager/0.log"
Jul 11 04:28:36.659790 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:36.659763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7x6sp_e1de9211-8287-44cf-9608-a949ba0ad8b4/kuadrant-console-plugin/0.log"
Jul 11 04:28:36.773073 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:36.773049    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6ghz9_69f17c55-8391-483d-9a85-8d8014b2eb87/registry-server/0.log"
Jul 11 04:28:36.886106 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:36.886076    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7vzfz_0f467436-7dfc-445b-b0ec-41781e1f5047/manager/0.log"
Jul 11 04:28:37.005739 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:37.005713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jsnps_3303dbfa-b6f9-4075-80fe-7e4546bbe087/limitador/0.log"
Jul 11 04:28:37.803469 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:37.803432    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67cfbd58d5-st9mj_43a572d7-0033-4845-be73-b514a5121669/kube-auth-proxy/0.log"
Jul 11 04:28:37.923074 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:37.923044    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/1.log"
Jul 11 04:28:38.156799 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:38.156713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-crhmt_10a64279-544a-4e5f-a57c-43f3bf17b598/payload-processing/0.log"
Jul 11 04:28:38.628683 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:38.628653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-66g6l_d675d7d0-9d4c-4121-a0df-824ebb60ff7e/main/0.log"
Jul 11 04:28:38.959904 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:38.959820    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzhfcm_0400edf2-cc87-4ee3-a035-1c91b08c22e0/main/0.log"
Jul 11 04:28:39.183234 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:39.183204    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xwpkz_1e7024fd-fc12-4baa-8e93-5eb9b0d9f114/main/0.log"
Jul 11 04:28:43.090215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:43.090181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:43.151285 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:43.151248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:45.833337 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:45.833308    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wgr8c_5e41e790-86e5-4643-8afc-2f49245db72e/global-pull-secret-syncer/0.log"
Jul 11 04:28:45.877070 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:45.877044    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log"
Jul 11 04:28:45.987672 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:45.987644    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-230.ec2.internal_32298df727bbd8ffb50d4d225716d8ff/haproxy/0.log"
Jul 11 04:28:50.760559 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:50.760528    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-79jz4_650de894-ccd5-4088-843e-0fbb21075709/manager/0.log"
Jul 11 04:28:50.784659 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:50.784633    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kswfr_315db466-92b7-423c-99ef-a6f2c0459199/manager/0.log"
Jul 11 04:28:50.810048 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:50.810024    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7x6sp_e1de9211-8287-44cf-9608-a949ba0ad8b4/kuadrant-console-plugin/0.log"
Jul 11 04:28:50.846948 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:50.846917    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6ghz9_69f17c55-8391-483d-9a85-8d8014b2eb87/registry-server/0.log"
Jul 11 04:28:50.995785 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:50.995752    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7vzfz_0f467436-7dfc-445b-b0ec-41781e1f5047/manager/0.log"
Jul 11 04:28:51.099730 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:51.099651    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jsnps_3303dbfa-b6f9-4075-80fe-7e4546bbe087/limitador/0.log"
Jul 11 04:28:53.224087 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:53.224052    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqdmt_cf126a7e-3174-4727-a28e-27e3e6985488/node-exporter/0.log"
Jul 11 04:28:53.256020 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:53.255992    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqdmt_cf126a7e-3174-4727-a28e-27e3e6985488/kube-rbac-proxy/0.log"
Jul 11 04:28:53.276947 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:53.276922    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqdmt_cf126a7e-3174-4727-a28e-27e3e6985488/init-textfile/0.log"
Jul 11 04:28:55.394365 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:55.394333    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-tsrsg_60537793-ea91-4fac-aa1b-a1026778eead/networking-console-plugin/0.log"
Jul 11 04:28:56.294785 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.294744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:56.348473 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.348437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hfzf5_aeb18047-4145-49e9-844e-4bdb4f0f8eb0/konnectivity-agent/0.log" limit=1024
Jul 11 04:28:56.470268 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470183    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"]
Jul 11 04:28:56.470766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470622    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a73028c-6ffa-437d-a782-f9e69fc88d2b" containerName="revoke-keys"
Jul 11 04:28:56.470766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470636    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a73028c-6ffa-437d-a782-f9e69fc88d2b" containerName="revoke-keys"
Jul 11 04:28:56.470766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470652    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f744fe7-7e56-41c8-8704-0b829ddb2044" containerName="maas-api"
Jul 11 04:28:56.470766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470657    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f744fe7-7e56-41c8-8704-0b829ddb2044" containerName="maas-api"
Jul 11 04:28:56.470766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470724    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f744fe7-7e56-41c8-8704-0b829ddb2044" containerName="maas-api"
Jul 11 04:28:56.470766 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.470734    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a73028c-6ffa-437d-a782-f9e69fc88d2b" containerName="revoke-keys"
Jul 11 04:28:56.473966 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.473948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.476542 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.476485    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cfvxb\"/\"default-dockercfg-tsckk\""
Jul 11 04:28:56.479140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.477190    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfvxb\"/\"openshift-service-ca.crt\""
Jul 11 04:28:56.479140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.477519    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfvxb\"/\"kube-root-ca.crt\""
Jul 11 04:28:56.486381 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.485678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"]
Jul 11 04:28:56.590960 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.590883    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68f7cc5864-dl6b5_4a3447dc-15e1-402d-9926-156ab9c304c1/console/0.log"
Jul 11 04:28:56.599338 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.599310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-lib-modules\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.599482 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.599351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x876p\" (UniqueName: \"kubernetes.io/projected/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-kube-api-access-x876p\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.599482 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.599418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-podres\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.599607 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.599500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-sys\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.599607 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.599573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-proc\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700307 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-proc\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700502 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-lib-modules\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700502 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x876p\" (UniqueName: \"kubernetes.io/projected/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-kube-api-access-x876p\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700502 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-podres\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700502 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-proc\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700502 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-sys\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-lib-modules\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-sys\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.700716 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.700572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-podres\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.707590 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.707565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x876p\" (UniqueName: \"kubernetes.io/projected/d50de85a-0e5d-48dc-b5ad-847fa2810aa0-kube-api-access-x876p\") pod \"perf-node-gather-daemonset-99cp9\" (UID: \"d50de85a-0e5d-48dc-b5ad-847fa2810aa0\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.787605 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.787568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:56.930393 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:56.930367    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"]
Jul 11 04:28:56.932273 ip-10-0-140-230 kubenswrapper[2582]: W0711 04:28:56.932244    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd50de85a_0e5d_48dc_b5ad_847fa2810aa0.slice/crio-5f8a52aee4c133538bc4b336b10ee6c418dfd7bf86e400d2b9159129b2068bdb WatchSource:0}: Error finding container 5f8a52aee4c133538bc4b336b10ee6c418dfd7bf86e400d2b9159129b2068bdb: Status 404 returned error can't find the container with id 5f8a52aee4c133538bc4b336b10ee6c418dfd7bf86e400d2b9159129b2068bdb
Jul 11 04:28:57.213331 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:57.213253    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-vz9z4_b316cf50-eb33-4e1c-85a1-d800cb5ce977/volume-data-source-validator/0.log"
Jul 11 04:28:57.311759 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:57.311724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9" event={"ID":"d50de85a-0e5d-48dc-b5ad-847fa2810aa0","Type":"ContainerStarted","Data":"a8dcb7de1f697bb9927b9bb45f03bfcbfd4112dd8777647f3114138def9cb7a2"}
Jul 11 04:28:57.311934 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:57.311773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9" event={"ID":"d50de85a-0e5d-48dc-b5ad-847fa2810aa0","Type":"ContainerStarted","Data":"5f8a52aee4c133538bc4b336b10ee6c418dfd7bf86e400d2b9159129b2068bdb"}
Jul 11 04:28:57.311934 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:57.311874    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:28:57.327975 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:57.327928    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9" podStartSLOduration=1.327913359 podStartE2EDuration="1.327913359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 04:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:28:57.325554492 +0000 UTC m=+2782.195001056" watchObservedRunningTime="2026-07-11 04:28:57.327913359 +0000 UTC m=+2782.197359922"
Jul 11 04:28:58.289293 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:58.289262    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzhr2_1bb1cce9-f75d-42b2-b5ad-e123906216c0/dns/0.log"
Jul 11 04:28:58.309083 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:58.309046    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzhr2_1bb1cce9-f75d-42b2-b5ad-e123906216c0/kube-rbac-proxy/0.log"
Jul 11 04:28:58.375271 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:58.375243    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rdtx4_21e8f34f-1131-4a4e-9830-7eba4d3135e4/dns-node-resolver/0.log"
Jul 11 04:28:58.944138 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:28:58.944084    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nswmp_eb2d92a6-4c87-47ae-af3f-8634e40903e6/node-ca/0.log"
Jul 11 04:29:00.014215 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:00.014185    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67cfbd58d5-st9mj_43a572d7-0033-4845-be73-b514a5121669/kube-auth-proxy/0.log"
Jul 11 04:29:00.044498 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:00.044462    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/0.log"
Jul 11 04:29:00.076297 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:00.076265    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/1.log"
Jul 11 04:29:00.134378 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:00.134341    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-crhmt_10a64279-544a-4e5f-a57c-43f3bf17b598/payload-processing/0.log"
Jul 11 04:29:00.886744 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:00.886717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxg8q_14abdefe-12c5-4266-a648-06bd2fb711e1/serve-healthcheck-canary/0.log"
Jul 11 04:29:01.373207 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:01.373170    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-crc28_c26922ba-a360-408d-a2ac-e150405b17b8/kube-rbac-proxy/0.log"
Jul 11 04:29:01.392564 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:01.392539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-crc28_c26922ba-a360-408d-a2ac-e150405b17b8/exporter/0.log"
Jul 11 04:29:01.416355 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:01.416324    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-crc28_c26922ba-a360-408d-a2ac-e150405b17b8/extractor/0.log"
Jul 11 04:29:03.327286 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.327259    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-99cp9"
Jul 11 04:29:03.593930 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.593847    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bffn4_ff10d8c5-3d3a-4935-a4f6-d193469e6e6c/manager/0.log"
Jul 11 04:29:03.732858 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.732820    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vd8l9_9a83a035-1fa4-4f07-9bf4-31be8f6e5661/manager/0.log"
Jul 11 04:29:03.829664 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.829625    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57c8d895cd-lmmpf_e03d8538-d22e-4b32-b4af-879a2d0b3704/manager/0.log"
Jul 11 04:29:03.924011 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.923923    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4hfnt_b29da052-ad41-4ff0-840e-9eccef268909/manager/0.log"
Jul 11 04:29:03.949843 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.949818    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-b8mwq_c7bd2c6b-7412-436a-a48c-a37b239480b8/manager/0.log"
Jul 11 04:29:03.969514 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:03.969489    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-b8mwq_c7bd2c6b-7412-436a-a48c-a37b239480b8/copy-manifests/0.log"
Jul 11 04:29:06.752973 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:06.752942    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67cfbd58d5-st9mj_43a572d7-0033-4845-be73-b514a5121669/kube-auth-proxy/0.log"
Jul 11 04:29:06.770630 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:06.770596    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/0.log"
Jul 11 04:29:06.798484 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:06.798455    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/1.log"
Jul 11 04:29:06.838328 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:06.838298    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-crhmt_10a64279-544a-4e5f-a57c-43f3bf17b598/payload-processing/0.log"
Jul 11 04:29:08.400110 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:08.400072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67cfbd58d5-st9mj_43a572d7-0033-4845-be73-b514a5121669/kube-auth-proxy/0.log"
Jul 11 04:29:08.419920 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:08.419892    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/0.log"
Jul 11 04:29:08.448615 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:08.448590    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-c4hrq_dc975611-2bb1-481c-b786-1399a8f95041/istio-proxy/1.log"
Jul 11 04:29:08.496724 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:08.496702    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-crhmt_10a64279-544a-4e5f-a57c-43f3bf17b598/payload-processing/0.log"
Jul 11 04:29:12.338612 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.338584    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/kube-multus-additional-cni-plugins/0.log"
Jul 11 04:29:12.358326 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.358301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/egress-router-binary-copy/0.log"
Jul 11 04:29:12.377270 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.377246    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/cni-plugins/0.log"
Jul 11 04:29:12.397371 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.397350    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/bond-cni-plugin/0.log"
Jul 11 04:29:12.416912 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.416882    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/routeoverride-cni/0.log"
Jul 11 04:29:12.436140 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.436117    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/whereabouts-cni-bincopy/0.log"
Jul 11 04:29:12.456454 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.456418    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlw6w_e0339b6f-0473-42e8-854f-c943ab81ec25/whereabouts-cni/0.log"
Jul 11 04:29:12.549190 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.549162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nlkqm_7e3fb74a-3bb1-45d1-94f4-c52eb1c5074c/kube-multus/0.log"
Jul 11 04:29:12.610962 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.610875    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qlm47_2be65b1e-0556-42b8-818d-e596d95d4a15/network-metrics-daemon/0.log"
Jul 11 04:29:12.627918 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:12.627895    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qlm47_2be65b1e-0556-42b8-818d-e596d95d4a15/kube-rbac-proxy/0.log"
Jul 11 04:29:13.470337 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.470308    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-controller/0.log"
Jul 11 04:29:13.485406 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.485381    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/0.log"
Jul 11 04:29:13.485540 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.485464    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovn-acl-logging/1.log"
Jul 11 04:29:13.501606 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.501585    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/kube-rbac-proxy-node/0.log"
Jul 11 04:29:13.521281 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.521257    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/kube-rbac-proxy-ovn-metrics/0.log"
Jul 11 04:29:13.539663 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.539641    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/northd/0.log"
Jul 11 04:29:13.558779 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.558751    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/nbdb/0.log"
Jul 11 04:29:13.577787 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.577767    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/sbdb/0.log"
Jul 11 04:29:13.631504 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:13.631473    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8r9wt_4e522d6f-2360-44a9-8d75-6013ab33d709/ovnkube-controller/0.log"
Jul 11 04:29:15.175336 ip-10-0-140-230 kubenswrapper[2582]: I0711 04:29:15.175305    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-p8hsg_ee95f72b-4aa3-4eb2-9770-24cdd7109fdb/check-endpoints/0.log"