Jul 09 14:39:58.866250 ip-10-0-142-236 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 14:39:59.213560 ip-10-0-142-236 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 14:39:59.213560 ip-10-0-142-236 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 14:39:59.213560 ip-10-0-142-236 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 14:39:59.213560 ip-10-0-142-236 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 14:39:59.213560 ip-10-0-142-236 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 14:39:59.214221 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.213557    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 14:39:59.217986 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.217959    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.217990    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.217996    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218001    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218006    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218010    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218014    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218017    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218021    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218024    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218028    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 14:39:59.218030 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218033    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218037    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218041    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218045    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218048    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218052    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218055    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218059    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218063    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218066    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218071    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218074    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218078    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218082    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218086    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218090    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218094    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218097    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218101    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 14:39:59.218294 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218104    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218111    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218117    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218121    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218125    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218129    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218133    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218138    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218141    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218145    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218149    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218152    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218158    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218163    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218167    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218172    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218176    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218179    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218183    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 14:39:59.218942 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218187    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218192    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218208    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218212    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218216    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218219    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218223    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218226    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218230    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218234    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218237    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218242    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218245    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218249    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218252    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218255    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 14:39:59.219483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.218298    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 14:39:59.220314 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220303    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 14:39:59.220314 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220314    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220320    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220327    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220333    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220337    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220345    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220350    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220354    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220359    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220363    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220372    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 14:39:59.220375 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220376    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220381    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220390    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220394    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220399    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220403    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220407    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220411    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220415    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220419    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220423    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220427    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220431    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220435    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220439    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220447    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220451    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220455    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220459    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220463    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 14:39:59.220749 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.220478    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 14:39:59.221505 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221488    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 14:39:59.221505 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221505    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221509    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221512    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221514    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221517    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221519    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221522    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221525    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221527    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221529    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221532    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221534    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221537    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221540    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221542    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221545    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221548    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221551    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221555    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 14:39:59.221553 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221559    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221562    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221565    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221568    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221571    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221573    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221576    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221579    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221582    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221584    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221587    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221589    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221593    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221598    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221601    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221603    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221606    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221609    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221612    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221614    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 14:39:59.222017 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221617    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221619    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221622    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221624    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221626    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221629    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221631    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221634    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221636    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221639    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221642    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221645    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221647    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221650    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221656    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221658    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221661    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221663    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221666    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221668    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 14:39:59.222479 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221671    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221674    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221676    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221679    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221681    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221684    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221686    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221689    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221691    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221694    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221696    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221699    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221701    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221704    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221706    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221709    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221711    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221714    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221717    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221719    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 14:39:59.222936 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221722    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221724    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221727    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221730    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221732    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221735    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221739    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221741    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221745    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221748    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221751    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221754    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221756    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221759    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221761    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221764    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221766    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221769    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.221771    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 14:39:59.223451 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222154    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222159    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222162    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222165    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222167    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222170    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222173    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222176    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222178    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222181    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222184    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222187    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222191    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222194    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222197    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222200    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222202    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222205    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222207    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 14:39:59.223907 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222212    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222214    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222217    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222219    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222222    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222225    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222228    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222230    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222232    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222235    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222237    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222240    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222242    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222245    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222247    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222250    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222252    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222255    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222257    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222260    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 14:39:59.224376 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222262    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222265    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222267    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222269    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222272    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222274    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222277    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222279    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222281    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222284    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222288    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222290    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222293    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222296    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222300    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222303    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222306    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222309    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222311    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222314    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 14:39:59.224858 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222316    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222319    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222321    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222324    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222327    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222329    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222331    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222334    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222336    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222339    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222341    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222345    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222349    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222353    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222355    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222358    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222361    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222363    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222366    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222368    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 14:39:59.225342 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222371    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222373    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222376    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222379    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222383    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222385    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222388    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222390    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222392    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222395    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222397    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222399    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222402    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222405    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222407    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222409    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222412    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222414    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222417    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.222420    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222501    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 14:39:59.225874 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222511    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222520    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222525    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222531    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222535    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222541    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222545    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222549    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222552    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222555    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222558    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222561    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222564    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222567    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222570    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222574    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222577    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222582    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222585    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222587    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222590    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222594    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222598    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 14:39:59.226395 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222600    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222603    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222607    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222609    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222612    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222615    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222618    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222621    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222625    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222628    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222631    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222634    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222637    2578 flags.go:64] FLAG: --enable-server="true"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222640    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222644    2578 flags.go:64] FLAG: --event-burst="100"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222647    2578 flags.go:64] FLAG: --event-qps="50"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222649    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222652    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222655    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222659    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222661    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222664    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222667    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222670    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222674    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 14:39:59.226924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222677    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222680    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222683    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222686    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222688    2578 flags.go:64] FLAG: --feature-gates=""
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222692    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222695    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222698    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222701    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222704    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222706    2578 flags.go:64] FLAG: --help="false"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222709    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222712    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222715    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222718    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222722    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222725    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222728    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222731    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222734    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222736    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222739    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222742    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222745    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 09 14:39:59.227548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222747    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222750    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222753    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222756    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222759    2578 flags.go:64] FLAG: --lock-file=""
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222761    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222764    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222768    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222774    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222777    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222779    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222782    2578 flags.go:64] FLAG: --logging-format="text"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222785    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222788    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222795    2578 flags.go:64] FLAG: --manifest-url=""
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222798    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222802    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222805    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222809    2578 flags.go:64] FLAG: --max-pods="110"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222811    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222814    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222817    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222820    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222823    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222826    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 14:39:59.228138 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222829    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222836    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222839    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222842    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222845    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222847    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222853    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222855    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222858    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222861    2578 flags.go:64] FLAG: --port="10250"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222864    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222867    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00165be9d12d89c87"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222870    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222873    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222876    2578 flags.go:64] FLAG: --register-node="true"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222880    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222883    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222886    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222889    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222891    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222895    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222898    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222901    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222903    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222906    2578 flags.go:64] FLAG: --runonce="false"
Jul 09 14:39:59.228731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222909    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222912    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222915    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222917    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222920    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222923    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222926    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222929    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222931    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222934    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222937    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222940    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222943    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222945    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222948    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222953    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222956    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222959    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222963    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222965    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222968    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222971    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222986    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222989    2578 flags.go:64] FLAG: --v="2"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222994    2578 flags.go:64] FLAG: --version="false"
Jul 09 14:39:59.229398 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.222998    2578 flags.go:64] FLAG: --vmodule=""
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.223001    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.223004    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223114    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223120    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223123    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223126    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223129    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223132    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223134    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223137    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223141    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223144    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223146    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223149    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223151    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223154    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223157    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223159    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 14:39:59.230006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223162    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223164    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223167    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223169    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223171    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223174    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223177    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223179    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223182    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223184    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223189    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223192    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223195    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223197    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223200    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223202    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223205    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223208    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223210    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223212    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 14:39:59.230460 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223215    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223217    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223220    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223222    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223226    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223228    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223232    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223236    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223239    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223241    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223244    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223247    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223250    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223252    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223255    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223257    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223260    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223262    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223265    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 14:39:59.230931 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223267    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223269    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223272    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223276    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223278    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223287    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223290    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223293    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223296    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223298    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223301    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223304    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223306    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223309    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223311    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223314    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223317    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223339    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223342    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223344    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 14:39:59.231503 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223347    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223349    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223352    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223354    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223357    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223360    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223362    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223365    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223368    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223372    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223377    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223380    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223383    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223385    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223388    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223391    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223394    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223397    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223400    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223402    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 14:39:59.232006 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223406    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223410    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223412    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.223415    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.223420    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231042    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231058    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231072    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231077    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231145    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231150    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231153    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231156    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231159    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 14:39:59.232496 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231162    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231166    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231169    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231172    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231175    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231178    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231180    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231183    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231186    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231188    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231191    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231195    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231199    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231202    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231204    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231207    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231210    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231213    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231216    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 14:39:59.232838 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231219    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231221    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231224    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231227    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231229    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231233    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231235    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231238    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231241    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231244    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231246    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231249    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231251    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231254    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231257    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231259    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231262    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231265    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231267    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231270    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 14:39:59.233301 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231272    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231275    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231279    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231292    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231295    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231297    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231300    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231303    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231306    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231308    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231311    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231313    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231318    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231320    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231323    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231327    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231330    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231334    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231337    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 14:39:59.233764 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231340    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231343    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231345    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231348    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231350    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231353    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231355    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231358    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231361    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231363    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231366    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231368    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231371    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231373    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231376    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231379    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231381    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231383    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231386    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231389    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 14:39:59.234224 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231391    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231394    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231396    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231399    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231401    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231404    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231407    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231410    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231412    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231414    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231418    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231420    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231423    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231425    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231428    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 14:39:59.234688 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231430    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231435    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231564    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231570    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231573    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231575    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231579    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231581    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231585    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231589    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231592    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231595    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231598    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231600    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231603    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 14:39:59.235047 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231606    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231608    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231611    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231613    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231616    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231618    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231621    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231623    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231626    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231631    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231634    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231636    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231639    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231641    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231644    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231647    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231649    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231652    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231654    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231657    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 14:39:59.235474 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231659    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231661    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231664    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231666    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231669    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231671    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231674    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231677    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231679    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231681    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231684    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231686    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231689    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231691    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231694    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231697    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231699    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231701    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231704    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231707    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 14:39:59.235955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231709    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231713    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231717    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231720    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231722    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231725    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231728    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231730    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231733    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231735    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231739    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231742    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231745    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231748    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231750    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231753    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231755    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231758    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231761    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 14:39:59.236531 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231764    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231766    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231768    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231771    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231774    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231776    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231779    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231782    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231784    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231787    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231789    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231792    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231794    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231797    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231799    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231802    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231808    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231810    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231813    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231815    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 14:39:59.237003 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231818    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231820    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231823    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231825    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231828    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231830    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:39:59.231833    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231837    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.231929    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.233968    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.235161    2578 server.go:1019] "Starting client certificate rotation"
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.235263    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.235311    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 14:39:59.237506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.237308    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 14:39:59.240246 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.240231    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 14:39:59.248187 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.248164    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 09 14:39:59.250220 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.250206    2578 log.go:25] "Validated CRI v1 image API"
Jul 09 14:39:59.251006 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.250971    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 14:39:59.253313 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.253291    2578 fs.go:135] Filesystem UUIDs: map[2b767d26-42ad-46be-b7e4-4fc2b93e506b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a0102f4e-120f-4c9f-8b75-9017e67c118e:/dev/nvme0n1p3]
Jul 09 14:39:59.253388 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.253310    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 14:39:59.258334 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.258216    2578 manager.go:216] Machine: {Timestamp:2026-07-09 14:39:59.257103087 +0000 UTC m=+0.290631560 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092278 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f656ae0fe0a41a964a4cb8327e317 SystemUUID:ec2f656a-e0fe-0a41-a964-a4cb8327e317 BootID:0622abed-ba9c-489b-8501-aabbdf910545 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:e1:ec:39:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:e1:ec:39:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:0f:51:9f:70:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 14:39:59.258334 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.258323    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 14:39:59.258485 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.258393    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 14:39:59.260966 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.260937    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 14:39:59.261099 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.260967    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 14:39:59.262102 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.262090    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 14:39:59.262102 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.262102    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 14:39:59.262184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.262109    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 14:39:59.262692 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.262682    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 14:39:59.263989 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.263966    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 14:39:59.266969 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.266951    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 14:39:59.267713 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267700    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 14:39:59.267831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267821    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 14:39:59.267873 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267851    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 14:39:59.267873 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267866    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 14:39:59.267945 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267881    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 14:39:59.267945 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267890    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 09 14:39:59.267945 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.267898    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 14:39:59.268254 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.268243    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 14:39:59.268294 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.268262    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 14:39:59.271563 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.271547    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 14:39:59.273309 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273296    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 14:39:59.273345 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273325    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 14:39:59.273475 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273466    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 14:39:59.273514 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273480    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 14:39:59.273514 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273486    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 14:39:59.273514 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273492    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 14:39:59.273514 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273499    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 14:39:59.273514 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273504    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 14:39:59.273957 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273940    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 14:39:59.273957 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273956    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 14:39:59.274061 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273963    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 14:39:59.274061 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.273969    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 14:39:59.274061 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.274000    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 14:39:59.274061 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.274011    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 14:39:59.274667 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.274655    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 14:39:59.274699 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.274669    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 14:39:59.277579 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.277566    2578 server.go:1268] "Started kubelet"
Jul 09 14:39:59.277730 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.277657    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 14:39:59.277730 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.277664    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 14:39:59.277822 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.277752    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 14:39:59.278506 ip-10-0-142-236 systemd[1]: Started Kubernetes Kubelet.
Jul 09 14:39:59.278951 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.278927    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 14:39:59.280204 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.280192    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 14:39:59.280625 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.280580    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 14:39:59.280698 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.280622    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 14:39:59.280789 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.280644    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 14:39:59.283942 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.283923    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 14:39:59.284427 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.283616    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18c0a6067f7b96d3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:ip-10-0-142-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-07-09 14:39:59.277536979 +0000 UTC m=+0.311065453,LastTimestamp:2026-07-09 14:39:59.277536979 +0000 UTC m=+0.311065453,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}"
Jul 09 14:39:59.284829 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.284815    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 14:39:59.284876 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.284828    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 14:39:59.284920 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.284912    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.285495 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285473    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 14:39:59.285495 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285493    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 14:39:59.285631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285506    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 14:39:59.285631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285514    2578 factory.go:55] Registering systemd factory
Jul 09 14:39:59.285631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285529    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 09 14:39:59.285631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285595    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 14:39:59.285631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285603    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 14:39:59.285895 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285719    2578 factory.go:153] Registering CRI-O factory
Jul 09 14:39:59.285895 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285734    2578 factory.go:223] Registration of the crio container factory successfully
Jul 09 14:39:59.285895 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285781    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 14:39:59.285895 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285802    2578 factory.go:103] Registering Raw factory
Jul 09 14:39:59.285895 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.285813    2578 manager.go:1196] Started watching for new ooms in manager
Jul 09 14:39:59.286175 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.286163    2578 manager.go:349] Starting recovery of all containers
Jul 09 14:39:59.291648 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.291616    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 14:39:59.291776 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.291756    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 14:39:59.296053 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.296041    2578 manager.go:354] Recovery completed
Jul 09 14:39:59.300611 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.300600    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.302304 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302280    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.302371 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302317    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.302371 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302330    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.302787 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302776    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 14:39:59.302825 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302788    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 14:39:59.302825 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302802    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 14:39:59.302905 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.302780    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.303572 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.303558    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.303623 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.303579    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.303623 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.303589    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.304650 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.304597    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18c0a60680f57b93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:ip-10-0-142-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-07-09 14:39:59.302302611 +0000 UTC m=+0.335831086,LastTimestamp:2026-07-09 14:39:59.302302611 +0000 UTC m=+0.335831086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}"
Jul 09 14:39:59.305001 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.304986    2578 policy_none.go:49] "None policy: Start"
Jul 09 14:39:59.305049 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.305005    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 14:39:59.305049 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.305017    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 14:39:59.307052 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.307040    2578 policy_none.go:47] "Start"
Jul 09 14:39:59.316028 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.315931    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18c0a60680f5cd83  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:ip-10-0-142-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-236.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-07-09 14:39:59.302323587 +0000 UTC m=+0.335852063,LastTimestamp:2026-07-09 14:39:59.302323587 +0000 UTC m=+0.335852063,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}"
Jul 09 14:39:59.317714 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.317699    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k4w6k"
Jul 09 14:39:59.326017 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.325936    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18c0a60680f5f620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:ip-10-0-142-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-236.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-07-09 14:39:59.302333984 +0000 UTC m=+0.335862457,LastTimestamp:2026-07-09 14:39:59.302333984 +0000 UTC m=+0.335862457,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}"
Jul 09 14:39:59.331704 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.331591    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k4w6k"
Jul 09 14:39:59.340595 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.340583    2578 manager.go:337] "Starting Device Plugin manager"
Jul 09 14:39:59.340674 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.340610    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 14:39:59.340674 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.340619    2578 server.go:87] "Starting device plugin registration server"
Jul 09 14:39:59.340842 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.340830    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 14:39:59.340890 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.340843    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 14:39:59.340939 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.340925    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 14:39:59.341037 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.341020    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 14:39:59.341037 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.341035    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 14:39:59.341411 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.341391    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 14:39:59.341509 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.341442    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.441345 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.441323    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.443342 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.442105    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.443342 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.442135    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.443342 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.442149    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.443342 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.442173    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.450729 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.450716    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.450793 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.450735    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-236.ec2.internal\": node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.475149 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.475102    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.481438 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.481419    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 14:39:59.482535 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.482518    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 14:39:59.482590 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.482562    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 14:39:59.482590 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.482586    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 14:39:59.482675 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.482622    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 14:39:59.484678 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.484660    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 14:39:59.576200 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.576172    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.583345 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.583319    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"]
Jul 09 14:39:59.583407 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.583379    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.584759 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.584739    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.584837 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.584766    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.584837 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.584776    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.587042 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a76ec413de110e8f73baffdec3c3161-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"1a76ec413de110e8f73baffdec3c3161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.587096 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a76ec413de110e8f73baffdec3c3161-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"1a76ec413de110e8f73baffdec3c3161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.587096 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587069    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.587179 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.587223 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587197    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.587782 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587766    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.587782 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587782    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.587887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587794    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.587887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587803    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.587887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587807    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.587887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.587813    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.590116 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.590100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.590171 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.590130    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 14:39:59.591409 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.591394    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 14:39:59.591481 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.591419    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 14:39:59.591481 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.591429    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Jul 09 14:39:59.610543 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.610528    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.614346 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.614331    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.677058 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.677044    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.687731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.687714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a76ec413de110e8f73baffdec3c3161-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"1a76ec413de110e8f73baffdec3c3161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.687796 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.687742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d92aa44007fbeeba58b77b0f3aaedc2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"6d92aa44007fbeeba58b77b0f3aaedc2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.687796 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.687761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a76ec413de110e8f73baffdec3c3161-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"1a76ec413de110e8f73baffdec3c3161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.687869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.687799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a76ec413de110e8f73baffdec3c3161-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"1a76ec413de110e8f73baffdec3c3161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.687869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.687813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a76ec413de110e8f73baffdec3c3161-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"1a76ec413de110e8f73baffdec3c3161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.777691 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.777639    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.788841 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.788824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d92aa44007fbeeba58b77b0f3aaedc2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"6d92aa44007fbeeba58b77b0f3aaedc2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.788892 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.788862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d92aa44007fbeeba58b77b0f3aaedc2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"6d92aa44007fbeeba58b77b0f3aaedc2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.883720 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.883692    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:39:59.913358 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.913339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.917014 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:39:59.916997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Jul 09 14:39:59.985680 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:39:59.985651    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:40:00.092164 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:00.092113    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:40:00.193153 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:00.193128    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:40:00.234652 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.234618    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 14:40:00.235074 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.234806    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 14:40:00.293420 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:00.293397    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Jul 09 14:40:00.304450 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.304429    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 14:40:00.337015 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.336962    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 14:35:00 +0000 UTC" deadline="2028-01-15 05:33:36.181876318 +0000 UTC"
Jul 09 14:40:00.337015 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.337007    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13310h53m35.844872252s"
Jul 09 14:40:00.388144 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.388093    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Jul 09 14:40:00.403292 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.403273    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 14:40:00.404936 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.404923    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Jul 09 14:40:00.415295 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.415279    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 14:40:00.433385 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.433368    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 14:40:00.837456 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:00.837421    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a76ec413de110e8f73baffdec3c3161.slice/crio-9b74d900f95e68a866129414589fd12ef528160793c76dfddb117039664908f4 WatchSource:0}: Error finding container 9b74d900f95e68a866129414589fd12ef528160793c76dfddb117039664908f4: Status 404 returned error can't find the container with id 9b74d900f95e68a866129414589fd12ef528160793c76dfddb117039664908f4
Jul 09 14:40:00.843883 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.843857    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 14:40:00.849163 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:00.849142    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d92aa44007fbeeba58b77b0f3aaedc2.slice/crio-62d8363761541bc28666b54c35201e3b9d2519a6cbee198766b65719e1e22d04 WatchSource:0}: Error finding container 62d8363761541bc28666b54c35201e3b9d2519a6cbee198766b65719e1e22d04: Status 404 returned error can't find the container with id 62d8363761541bc28666b54c35201e3b9d2519a6cbee198766b65719e1e22d04
Jul 09 14:40:00.863661 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:00.863643    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 14:40:01.181248 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.181170    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 14:40:01.271201 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.271165    2578 apiserver.go:52] "Watching apiserver"
Jul 09 14:40:01.281204 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.281183    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 14:40:01.282475 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.282454    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-n7r6j","openshift-multus/multus-v22jf","openshift-network-operator/iptables-alerter-xgjt6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt","openshift-cluster-node-tuning-operator/tuned-2ld92","openshift-image-registry/node-ca-psqp4","openshift-multus/network-metrics-daemon-7vp7b","openshift-network-diagnostics/network-check-target-jnx8p","openshift-ovn-kubernetes/ovnkube-node-jr7vg","kube-system/global-pull-secret-syncer-w8jb7","kube-system/konnectivity-agent-c5htq","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal","openshift-dns/node-resolver-vvm8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"]
Jul 09 14:40:01.285361 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.285346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.285429 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.285410    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:01.286294 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.286274    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 14:40:01.289337 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.289274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.290478 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.290460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.292441 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.292423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d8hhh\""
Jul 09 14:40:01.292539 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.292445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.292820 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.292794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.292921 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.292874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.293901 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.293884    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 14:40:01.293997 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.293884    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.293997 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.293915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g4h5c\""
Jul 09 14:40:01.293997 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.293886    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.295368 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.295351    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.295682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.295658    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zbz4v\""
Jul 09 14:40:01.295761 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.295710    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.295829 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.295762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.295888 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.295864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 14:40:01.298105 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.297902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.298178 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.298129    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.298178 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.298140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9dqsr\""
Jul 09 14:40:01.298628 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.298606    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 14:40:01.298697 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.298616    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.299583 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.299563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.299642 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.299600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chm2z\" (UniqueName: \"kubernetes.io/projected/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-kube-api-access-chm2z\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.299642 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.299626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f93dcb3-a544-417e-839f-fb1fc4c85494-hosts-file\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.299752 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.299649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f93dcb3-a544-417e-839f-fb1fc4c85494-tmp-dir\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.299752 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.299674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkjf\" (UniqueName: \"kubernetes.io/projected/4f93dcb3-a544-417e-839f-fb1fc4c85494-kube-api-access-xfkjf\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.301005 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.300868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 14:40:01.301005 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.300940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.301005 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.300962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.301201 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.301175    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.301793 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.301773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pnmf4\""
Jul 09 14:40:01.303455 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.303437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 14:40:01.303801 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.303708    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 14:40:01.303931 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.303914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.304635 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.304614    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.305542 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.305529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6h7mm\""
Jul 09 14:40:01.305777 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.305756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 14:40:01.305873 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.305815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.305930 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.305885    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 14:40:01.306835 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.306816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.306977 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.306938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wcdb9\""
Jul 09 14:40:01.307082 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.307062    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 14:40:01.307469 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.307450    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 14:40:01.307597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.307470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 14:40:01.307597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.307505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 14:40:01.307597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.307555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 14:40:01.309467 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.309449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.309562 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.309531    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:01.309964 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.309947    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 14:40:01.310400 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.310381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-92dm4\""
Jul 09 14:40:01.312083 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.312065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.314856 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.314836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:01.315007 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.314986    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:01.315100 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.315016    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 14:40:01.315161 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.315145    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 14:40:01.315463 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.315449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sqv4r\""
Jul 09 14:40:01.331803 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.331776    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cp9z6"
Jul 09 14:40:01.337677 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.337654    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cp9z6"
Jul 09 14:40:01.386418 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.386399    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 14:40:01.400315 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-log-socket\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.400392 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-cni-netd\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.400444 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-ovnkube-script-lib\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.400444 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-cni-multus\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400517 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-daemon-config\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400517 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.400517 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bkv\" (UniqueName: \"kubernetes.io/projected/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-kube-api-access-b5bkv\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.400517 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-slash\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.400517 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-var-lib-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-etc-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-socket-dir-parent\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-cni-bin\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-etc-kubernetes\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f93dcb3-a544-417e-839f-fb1fc4c85494-tmp-dir\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-tuned\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-os-release\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-hostroot\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.400682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-conf-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20081f0a-4efc-4d49-941b-e5f0713855d3-agent-certs\") pod \"konnectivity-agent-c5htq\" (UID: \"20081f0a-4efc-4d49-941b-e5f0713855d3\") " pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01f91055-f6eb-47b4-8958-65fe57ea065a-host-slash\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f93dcb3-a544-417e-839f-fb1fc4c85494-tmp-dir\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.400933    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.400972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysconfig\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.401025    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:01.900987356 +0000 UTC m=+2.934515840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:01.401093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chm2z\" (UniqueName: \"kubernetes.io/projected/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-kube-api-access-chm2z\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-env-overrides\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4mm\" (UniqueName: \"kubernetes.io/projected/a20a5f44-66dd-4275-8807-af25b9282aaf-kube-api-access-lf4mm\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-system-cni-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2s7\" (UniqueName: \"kubernetes.io/projected/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-kube-api-access-tt2s7\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-modprobe-d\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-systemd\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcl5\" (UniqueName: \"kubernetes.io/projected/01f91055-f6eb-47b4-8958-65fe57ea065a-kube-api-access-nzcl5\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-run-netns\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-ovn\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-cni-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-cni-binary-copy\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-netns\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-os-release\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-cnibin\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-systemd-units\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-sys\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-k8s-cni-cncf-io\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-kubelet-config\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20081f0a-4efc-4d49-941b-e5f0713855d3-konnectivity-ca\") pod \"konnectivity-agent-c5htq\" (UID: \"20081f0a-4efc-4d49-941b-e5f0713855d3\") " pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/01f91055-f6eb-47b4-8958-65fe57ea065a-iptables-alerter-script\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-cnibin\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/886304ba-ee5f-4af4-9435-3a256bc1ede6-host\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-host\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-kubelet\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-kubernetes\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.401948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-systemd\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc82k\" (UniqueName: \"kubernetes.io/projected/886304ba-ee5f-4af4-9435-3a256bc1ede6-kube-api-access-fc82k\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-lib-modules\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-tmp\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-kubelet\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-multus-certs\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.401957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-registration-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-var-lib-kubelet\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-cni-bin\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-sys-fs\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h7w\" (UniqueName: \"kubernetes.io/projected/2a83c9e4-8a68-4de6-8720-e74e360e4610-kube-api-access-d6h7w\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f93dcb3-a544-417e-839f-fb1fc4c85494-hosts-file\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/886304ba-ee5f-4af4-9435-3a256bc1ede6-serviceca\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-system-cni-dir\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4f93dcb3-a544-417e-839f-fb1fc4c85494-hosts-file\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.402480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysctl-conf\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a20a5f44-66dd-4275-8807-af25b9282aaf-ovn-node-metrics-cert\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-dbus\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-socket-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkjf\" (UniqueName: \"kubernetes.io/projected/4f93dcb3-a544-417e-839f-fb1fc4c85494-kube-api-access-xfkjf\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysctl-d\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-device-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-node-log\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-ovnkube-config\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvcg\" (UniqueName: \"kubernetes.io/projected/9df2893b-e304-464d-a710-8d7094a8e4d2-kube-api-access-hvvcg\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-run\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.402988 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.402530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.408372 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.408352    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 14:40:01.411657 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.411628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chm2z\" (UniqueName: \"kubernetes.io/projected/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-kube-api-access-chm2z\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.411734 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.411628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkjf\" (UniqueName: \"kubernetes.io/projected/4f93dcb3-a544-417e-839f-fb1fc4c85494-kube-api-access-xfkjf\") pod \"node-resolver-vvm8k\" (UID: \"4f93dcb3-a544-417e-839f-fb1fc4c85494\") " pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.488585 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.488511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"6d92aa44007fbeeba58b77b0f3aaedc2","Type":"ContainerStarted","Data":"62d8363761541bc28666b54c35201e3b9d2519a6cbee198766b65719e1e22d04"}
Jul 09 14:40:01.489541 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.489521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"1a76ec413de110e8f73baffdec3c3161","Type":"ContainerStarted","Data":"9b74d900f95e68a866129414589fd12ef528160793c76dfddb117039664908f4"}
Jul 09 14:40:01.503640 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01f91055-f6eb-47b4-8958-65fe57ea065a-host-slash\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.503733 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysconfig\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.503733 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01f91055-f6eb-47b4-8958-65fe57ea065a-host-slash\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysconfig\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-env-overrides\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4mm\" (UniqueName: \"kubernetes.io/projected/a20a5f44-66dd-4275-8807-af25b9282aaf-kube-api-access-lf4mm\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-system-cni-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.503824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2s7\" (UniqueName: \"kubernetes.io/projected/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-kube-api-access-tt2s7\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-modprobe-d\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-system-cni-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.503961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-modprobe-d\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-systemd\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcl5\" (UniqueName: \"kubernetes.io/projected/01f91055-f6eb-47b4-8958-65fe57ea065a-kube-api-access-nzcl5\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.504184 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-systemd\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.504182    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-run-netns\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-run-netns\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.504336    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:02.00430395 +0000 UTC m=+3.037832429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-ovn\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-cni-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-ovn\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-cni-binary-copy\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-env-overrides\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-netns\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-os-release\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-cnibin\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-netns\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-os-release\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.504615 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-cni-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-systemd-units\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-cnibin\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-sys\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-systemd-units\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-k8s-cni-cncf-io\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-sys\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-k8s-cni-cncf-io\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-kubelet-config\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20081f0a-4efc-4d49-941b-e5f0713855d3-konnectivity-ca\") pod \"konnectivity-agent-c5htq\" (UID: \"20081f0a-4efc-4d49-941b-e5f0713855d3\") " pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/01f91055-f6eb-47b4-8958-65fe57ea065a-iptables-alerter-script\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-cnibin\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-kubelet-config\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/886304ba-ee5f-4af4-9435-3a256bc1ede6-host\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-cnibin\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-host\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-kubelet\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.505410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-host\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/886304ba-ee5f-4af4-9435-3a256bc1ede6-host\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.504986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-kubelet\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-kubernetes\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-systemd\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc82k\" (UniqueName: \"kubernetes.io/projected/886304ba-ee5f-4af4-9435-3a256bc1ede6-kube-api-access-fc82k\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-lib-modules\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-kubernetes\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-tmp\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-kubelet\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-multus-certs\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-kubelet\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-lib-modules\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.506333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-registration-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-systemd\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-var-lib-kubelet\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-cni-binary-copy\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-run-multus-certs\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-cni-bin\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-registration-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-sys-fs\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20081f0a-4efc-4d49-941b-e5f0713855d3-konnectivity-ca\") pod \"konnectivity-agent-c5htq\" (UID: \"20081f0a-4efc-4d49-941b-e5f0713855d3\") " pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-var-lib-kubelet\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h7w\" (UniqueName: \"kubernetes.io/projected/2a83c9e4-8a68-4de6-8720-e74e360e4610-kube-api-access-d6h7w\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/886304ba-ee5f-4af4-9435-3a256bc1ede6-serviceca\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-cni-bin\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-system-cni-dir\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-sys-fs\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysctl-conf\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.507268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df2893b-e304-464d-a710-8d7094a8e4d2-system-cni-dir\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a20a5f44-66dd-4275-8807-af25b9282aaf-ovn-node-metrics-cert\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysctl-conf\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-dbus\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-dbus\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-socket-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysctl-d\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-socket-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-device-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-node-log\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/886304ba-ee5f-4af4-9435-3a256bc1ede6-serviceca\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.505967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-sysctl-d\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a83c9e4-8a68-4de6-8720-e74e360e4610-device-dir\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-node-log\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-ovnkube-config\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/01f91055-f6eb-47b4-8958-65fe57ea065a-iptables-alerter-script\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.508019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvcg\" (UniqueName: \"kubernetes.io/projected/9df2893b-e304-464d-a710-8d7094a8e4d2-kube-api-access-hvvcg\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-run\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-run\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-log-socket\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-run-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-cni-netd\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-ovnkube-script-lib\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-cni-multus\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-log-socket\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-cni-netd\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-daemon-config\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-cni-multus\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bkv\" (UniqueName: \"kubernetes.io/projected/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-kube-api-access-b5bkv\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-slash\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.508597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-var-lib-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-etc-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-socket-dir-parent\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-cni-bin\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-etc-kubernetes\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-tuned\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-os-release\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-hostroot\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-conf-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.506986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20081f0a-4efc-4d49-941b-e5f0713855d3-agent-certs\") pod \"konnectivity-agent-c5htq\" (UID: \"20081f0a-4efc-4d49-941b-e5f0713855d3\") " pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-ovnkube-script-lib\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df2893b-e304-464d-a710-8d7094a8e4d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-daemon-config\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a20a5f44-66dd-4275-8807-af25b9282aaf-ovnkube-config\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-host-var-lib-cni-bin\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-os-release\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-etc-kubernetes\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-hostroot\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-var-lib-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-conf-dir\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-host-slash\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-multus-socket-dir-parent\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a20a5f44-66dd-4275-8807-af25b9282aaf-etc-openvswitch\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.507841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-tmp\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.509151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a20a5f44-66dd-4275-8807-af25b9282aaf-ovn-node-metrics-cert\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.509598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-etc-tuned\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.509910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.509799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20081f0a-4efc-4d49-941b-e5f0713855d3-agent-certs\") pod \"konnectivity-agent-c5htq\" (UID: \"20081f0a-4efc-4d49-941b-e5f0713855d3\") " pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.515962 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.515945    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:01.515962 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.515965    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:01.516071 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.515974    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:01.516071 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.516015    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:02.016004804 +0000 UTC m=+3.049533271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:01.517121 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.517098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcl5\" (UniqueName: \"kubernetes.io/projected/01f91055-f6eb-47b4-8958-65fe57ea065a-kube-api-access-nzcl5\") pod \"iptables-alerter-xgjt6\" (UID: \"01f91055-f6eb-47b4-8958-65fe57ea065a\") " pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.518598 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.518575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2s7\" (UniqueName: \"kubernetes.io/projected/2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5-kube-api-access-tt2s7\") pod \"multus-v22jf\" (UID: \"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5\") " pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.519094 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.519072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc82k\" (UniqueName: \"kubernetes.io/projected/886304ba-ee5f-4af4-9435-3a256bc1ede6-kube-api-access-fc82k\") pod \"node-ca-psqp4\" (UID: \"886304ba-ee5f-4af4-9435-3a256bc1ede6\") " pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.519203 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.519109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h7w\" (UniqueName: \"kubernetes.io/projected/2a83c9e4-8a68-4de6-8720-e74e360e4610-kube-api-access-d6h7w\") pod \"aws-ebs-csi-driver-node-pmlbt\" (UID: \"2a83c9e4-8a68-4de6-8720-e74e360e4610\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.519525 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.519510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4mm\" (UniqueName: \"kubernetes.io/projected/a20a5f44-66dd-4275-8807-af25b9282aaf-kube-api-access-lf4mm\") pod \"ovnkube-node-jr7vg\" (UID: \"a20a5f44-66dd-4275-8807-af25b9282aaf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.519889 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.519866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvcg\" (UniqueName: \"kubernetes.io/projected/9df2893b-e304-464d-a710-8d7094a8e4d2-kube-api-access-hvvcg\") pod \"multus-additional-cni-plugins-n7r6j\" (UID: \"9df2893b-e304-464d-a710-8d7094a8e4d2\") " pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.520149 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.520132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bkv\" (UniqueName: \"kubernetes.io/projected/35642d8c-98a6-496f-aa55-d96dc2ff0f1d-kube-api-access-b5bkv\") pod \"tuned-2ld92\" (UID: \"35642d8c-98a6-496f-aa55-d96dc2ff0f1d\") " pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.601909 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.601879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vvm8k"
Jul 09 14:40:01.608503 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.608490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xgjt6"
Jul 09 14:40:01.612280 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.612262    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f93dcb3_a544_417e_839f_fb1fc4c85494.slice/crio-882b083d6afd6b90740e6e7aa293033a634c4c9673a836f6f6082e36525b3f0b WatchSource:0}: Error finding container 882b083d6afd6b90740e6e7aa293033a634c4c9673a836f6f6082e36525b3f0b: Status 404 returned error can't find the container with id 882b083d6afd6b90740e6e7aa293033a634c4c9673a836f6f6082e36525b3f0b
Jul 09 14:40:01.617242 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.617221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt"
Jul 09 14:40:01.617394 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.617376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f91055_f6eb_47b4_8958_65fe57ea065a.slice/crio-60d169943c5290d17dbd6b5f9133956d7a4fa7b9daea658b27b1eda34526dc07 WatchSource:0}: Error finding container 60d169943c5290d17dbd6b5f9133956d7a4fa7b9daea658b27b1eda34526dc07: Status 404 returned error can't find the container with id 60d169943c5290d17dbd6b5f9133956d7a4fa7b9daea658b27b1eda34526dc07
Jul 09 14:40:01.620968 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.620949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2ld92"
Jul 09 14:40:01.627442 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.627421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-psqp4"
Jul 09 14:40:01.627813 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.627791    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a83c9e4_8a68_4de6_8720_e74e360e4610.slice/crio-96e8b22d9d004cf1c9808ba2ed1b4670a23f547d84e78f5b434bb086aa1a47a4 WatchSource:0}: Error finding container 96e8b22d9d004cf1c9808ba2ed1b4670a23f547d84e78f5b434bb086aa1a47a4: Status 404 returned error can't find the container with id 96e8b22d9d004cf1c9808ba2ed1b4670a23f547d84e78f5b434bb086aa1a47a4
Jul 09 14:40:01.630933 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.630904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35642d8c_98a6_496f_aa55_d96dc2ff0f1d.slice/crio-c73de704042700a4dad042b3134a74fa7493da567aa8a2400e07f0c19298d551 WatchSource:0}: Error finding container c73de704042700a4dad042b3134a74fa7493da567aa8a2400e07f0c19298d551: Status 404 returned error can't find the container with id c73de704042700a4dad042b3134a74fa7493da567aa8a2400e07f0c19298d551
Jul 09 14:40:01.633602 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.633570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:01.638543 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.638508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886304ba_ee5f_4af4_9435_3a256bc1ede6.slice/crio-a16870e5cb54e7ede920accba2d5af3c8cf62e9f5224ee7a57698285ca98396d WatchSource:0}: Error finding container a16870e5cb54e7ede920accba2d5af3c8cf62e9f5224ee7a57698285ca98396d: Status 404 returned error can't find the container with id a16870e5cb54e7ede920accba2d5af3c8cf62e9f5224ee7a57698285ca98396d
Jul 09 14:40:01.640704 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.640687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n7r6j"
Jul 09 14:40:01.643599 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.643578    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20a5f44_66dd_4275_8807_af25b9282aaf.slice/crio-d84637a24d08133fddaef08e024c7417262bd35c648d6e737ce9bec769ca5a5e WatchSource:0}: Error finding container d84637a24d08133fddaef08e024c7417262bd35c648d6e737ce9bec769ca5a5e: Status 404 returned error can't find the container with id d84637a24d08133fddaef08e024c7417262bd35c648d6e737ce9bec769ca5a5e
Jul 09 14:40:01.647746 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.647728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v22jf"
Jul 09 14:40:01.649067 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.649037    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df2893b_e304_464d_a710_8d7094a8e4d2.slice/crio-44dcf0a8594829984719a80829f126d8c44bda86ef60f827414dc26e0d72c340 WatchSource:0}: Error finding container 44dcf0a8594829984719a80829f126d8c44bda86ef60f827414dc26e0d72c340: Status 404 returned error can't find the container with id 44dcf0a8594829984719a80829f126d8c44bda86ef60f827414dc26e0d72c340
Jul 09 14:40:01.652476 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.652458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:01.656999 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.656977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe5c7d3_ffef_4c1c_a6f1_bda64e2895a5.slice/crio-c8e955c8d0de2547301e48da31ee5403fa8da4f533da72e1784fca15eb794f8f WatchSource:0}: Error finding container c8e955c8d0de2547301e48da31ee5403fa8da4f533da72e1784fca15eb794f8f: Status 404 returned error can't find the container with id c8e955c8d0de2547301e48da31ee5403fa8da4f533da72e1784fca15eb794f8f
Jul 09 14:40:01.661635 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:01.661617    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20081f0a_4efc_4d49_941b_e5f0713855d3.slice/crio-04dd03721fb10e35005663c1751811565ee8f15a52b84e5a133d0b96eadc611a WatchSource:0}: Error finding container 04dd03721fb10e35005663c1751811565ee8f15a52b84e5a133d0b96eadc611a: Status 404 returned error can't find the container with id 04dd03721fb10e35005663c1751811565ee8f15a52b84e5a133d0b96eadc611a
Jul 09 14:40:01.910602 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:01.910018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:01.910602 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.910193    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:01.910602 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:01.910254    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:02.91023528 +0000 UTC m=+3.943763746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:02.010482 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.010438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:02.010651 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.010562    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:02.010651 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.010623    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:03.010604467 +0000 UTC m=+4.044132940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:02.111360 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.111323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:02.111530 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.111508    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:02.111607 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.111534    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:02.111607 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.111548    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:02.111718 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.111621    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:03.111601665 +0000 UTC m=+4.145130130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:02.341780 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.341690    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:35:02 +0000 UTC" deadline="2027-12-17 17:49:41.943786385 +0000 UTC"
Jul 09 14:40:02.341780 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.341730    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12627h9m39.602061278s"
Jul 09 14:40:02.483705 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.483669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:02.483886 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.483822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:02.494032 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.493914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5htq" event={"ID":"20081f0a-4efc-4d49-941b-e5f0713855d3","Type":"ContainerStarted","Data":"04dd03721fb10e35005663c1751811565ee8f15a52b84e5a133d0b96eadc611a"}
Jul 09 14:40:02.496302 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.496232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v22jf" event={"ID":"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5","Type":"ContainerStarted","Data":"c8e955c8d0de2547301e48da31ee5403fa8da4f533da72e1784fca15eb794f8f"}
Jul 09 14:40:02.498662 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.498556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"d84637a24d08133fddaef08e024c7417262bd35c648d6e737ce9bec769ca5a5e"}
Jul 09 14:40:02.501627 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.501572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2ld92" event={"ID":"35642d8c-98a6-496f-aa55-d96dc2ff0f1d","Type":"ContainerStarted","Data":"c73de704042700a4dad042b3134a74fa7493da567aa8a2400e07f0c19298d551"}
Jul 09 14:40:02.504028 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.503947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt" event={"ID":"2a83c9e4-8a68-4de6-8720-e74e360e4610","Type":"ContainerStarted","Data":"96e8b22d9d004cf1c9808ba2ed1b4670a23f547d84e78f5b434bb086aa1a47a4"}
Jul 09 14:40:02.505700 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.505630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xgjt6" event={"ID":"01f91055-f6eb-47b4-8958-65fe57ea065a","Type":"ContainerStarted","Data":"60d169943c5290d17dbd6b5f9133956d7a4fa7b9daea658b27b1eda34526dc07"}
Jul 09 14:40:02.507269 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.507223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerStarted","Data":"44dcf0a8594829984719a80829f126d8c44bda86ef60f827414dc26e0d72c340"}
Jul 09 14:40:02.508574 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.508514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-psqp4" event={"ID":"886304ba-ee5f-4af4-9435-3a256bc1ede6","Type":"ContainerStarted","Data":"a16870e5cb54e7ede920accba2d5af3c8cf62e9f5224ee7a57698285ca98396d"}
Jul 09 14:40:02.510007 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.509963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vvm8k" event={"ID":"4f93dcb3-a544-417e-839f-fb1fc4c85494","Type":"ContainerStarted","Data":"882b083d6afd6b90740e6e7aa293033a634c4c9673a836f6f6082e36525b3f0b"}
Jul 09 14:40:02.920551 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:02.919899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:02.920551 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.920133    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:02.920551 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:02.920198    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:04.920179864 +0000 UTC m=+5.953708341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:03.021420 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:03.021384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:03.021600 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.021547    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:03.021683 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.021611    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:05.021592125 +0000 UTC m=+6.055120592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:03.121883 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:03.121840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:03.122157 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.122131    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:03.122254 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.122171    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:03.122254 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.122184    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:03.122445 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.122354    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:05.122286978 +0000 UTC m=+6.155815464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:03.342382 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:03.342281    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:35:02 +0000 UTC" deadline="2028-02-02 02:57:36.704302546 +0000 UTC"
Jul 09 14:40:03.342382 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:03.342326    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13740h17m33.361981168s"
Jul 09 14:40:03.483554 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:03.483520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:03.483743 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.483697    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:03.484240 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:03.484148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:03.484382 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:03.484252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:04.483920 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:04.483425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:04.483920 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:04.483554    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:04.937535 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:04.937445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:04.937715 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:04.937624    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:04.937715 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:04.937688    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:08.937669124 +0000 UTC m=+9.971197604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:05.038714 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:05.038077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:05.038714 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.038252    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:05.038714 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.038320    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:09.038300025 +0000 UTC m=+10.071828503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:05.139874 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:05.138492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:05.139874 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.138726    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:05.139874 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.138750    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:05.139874 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.138763    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:05.139874 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.138821    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:09.138802378 +0000 UTC m=+10.172330858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:05.488430 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:05.487236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:05.488430 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.487438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:05.488430 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:05.488159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:05.488430 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:05.488360    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:06.483657 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:06.483624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:06.483873 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:06.483785    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:07.483650 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:07.483616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:07.483650 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:07.483636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:07.484162 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:07.483765    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:07.484162 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:07.483877    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:08.483310 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:08.483275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:08.483479 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:08.483415    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:08.972180 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:08.971741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:08.972180 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:08.971915    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:08.972180 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:08.971989    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:16.97196897 +0000 UTC m=+18.005497448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:09.073055 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:09.072855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:09.073055 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.073008    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:09.073311 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.073096    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:17.073078074 +0000 UTC m=+18.106606560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:09.173485 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:09.173441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:09.173671 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.173642    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:09.173671 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.173665    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:09.173781 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.173681    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:09.173781 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.173739    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:17.173718916 +0000 UTC m=+18.207247445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:09.484561 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:09.484520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:09.484735 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.484660    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:09.484809 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:09.484730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:09.484862 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:09.484845    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:10.483468 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:10.483428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:10.483908 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:10.483597    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:11.483326 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:11.483242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:11.483487 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:11.483371    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:11.483872 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:11.483242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:11.483872 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:11.483819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:12.483571 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:12.483535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:12.484057 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:12.483668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:13.483594 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:13.483560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:13.484002 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:13.483658    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:13.484002 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:13.483748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:13.484002 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:13.483870    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:14.483525 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:14.483498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:14.483700 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:14.483612    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:15.482828 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:15.482792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:15.483042 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:15.482926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:15.483163 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:15.483131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:15.483271 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:15.483249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:16.482814 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:16.482774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:16.483245 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:16.482894    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:17.038826 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:17.038791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:17.039121 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.038923    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:17.039121 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.039005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:33.038971388 +0000 UTC m=+34.072499856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:17.139477 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:17.139449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:17.139661 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.139576    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:17.139661 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.139643    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:33.139624589 +0000 UTC m=+34.173153062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:17.240024 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:17.239994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:17.240199 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.240179    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:17.240267 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.240207    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:17.240267 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.240219    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:17.240365 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.240279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:33.240260255 +0000 UTC m=+34.273788730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:17.483127 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:17.483048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:17.483127 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:17.483091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:17.483539 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.483164    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:17.483539 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:17.483297    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:18.483268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:18.483226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:18.483684 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:18.483350    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:19.483945 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:19.483911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:19.484367 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:19.484036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:19.484367 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:19.484274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:19.484480 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:19.484397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:20.483035 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:20.483001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:20.483195 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:20.483104    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:21.483373 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:21.483341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:21.483831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:21.483400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:21.483831 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:21.483463    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:21.483831 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:21.483581    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:22.483043 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.482760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:22.483211 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:22.483114    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:22.554377 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.554299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vvm8k" event={"ID":"4f93dcb3-a544-417e-839f-fb1fc4c85494","Type":"ContainerStarted","Data":"3e1d2d254a9f985ca91b72c323aed1ff45d9f53219568fc2aa6a38ea5e0b85fd"}
Jul 09 14:40:22.555653 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.555626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"6d92aa44007fbeeba58b77b0f3aaedc2","Type":"ContainerStarted","Data":"b3c8eb7d6c570a75e53502ac4e5ffb598fbd6126d69db2500104be04a9d73a9f"}
Jul 09 14:40:22.556958 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.556934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5htq" event={"ID":"20081f0a-4efc-4d49-941b-e5f0713855d3","Type":"ContainerStarted","Data":"4aec0de3fb548ffccc799033938caf9512ed2ee10c981d67faaab72d9e78a292"}
Jul 09 14:40:22.558407 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.558382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v22jf" event={"ID":"2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5","Type":"ContainerStarted","Data":"1d020c16a45d1b51e7233dbf9b7368f1604faa828adf5fd845a3a93e40df5af0"}
Jul 09 14:40:22.561264 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561247    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:40:22.561580 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561561    2578 generic.go:356] "Generic (PLEG): container finished" podID="a20a5f44-66dd-4275-8807-af25b9282aaf" containerID="b008c2ccb3f3acf93b5061b7ad179dc1921e84195428d539428d2e5108d1be21" exitCode=1
Jul 09 14:40:22.561661 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"63be6b16202980c0b8488e0bfbaa198069e39777d560b9b38d4d0c4724aa256d"}
Jul 09 14:40:22.561731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"7437f8e61a28becd79aabfa695f7ea5291a75cdd81411e67596b73c26b875a2f"}
Jul 09 14:40:22.561775 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"8498a3e5abf6594262a315696f4dd827a70281c6935c46d8142b61384994774b"}
Jul 09 14:40:22.561823 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"59a09a0ed01f98b43f625936ac7057d237b0ca539aa242446d7d6aca33999312"}
Jul 09 14:40:22.561823 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerDied","Data":"b008c2ccb3f3acf93b5061b7ad179dc1921e84195428d539428d2e5108d1be21"}
Jul 09 14:40:22.561917 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.561846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"c70516f14467599c068d622eaae94f108e90923c47c34e2a20ff696b6bc833e3"}
Jul 09 14:40:22.563009 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.562963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2ld92" event={"ID":"35642d8c-98a6-496f-aa55-d96dc2ff0f1d","Type":"ContainerStarted","Data":"6657ae8f92b9e80efa7ebcb2221dfdd7d5a571c07594f86041af28825dc88581"}
Jul 09 14:40:22.564339 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.564319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt" event={"ID":"2a83c9e4-8a68-4de6-8720-e74e360e4610","Type":"ContainerStarted","Data":"eeeda35f22c6becd3f929a4938095e56506562005bd43ca1e8e7333bd018f7fc"}
Jul 09 14:40:22.565753 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.565722    2578 generic.go:356] "Generic (PLEG): container finished" podID="1a76ec413de110e8f73baffdec3c3161" containerID="a29f994bbb134ca951eaf257756ca42dadf607a9bf0bc8fc0ab7e16550edd250" exitCode=0
Jul 09 14:40:22.565753 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.565742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"1a76ec413de110e8f73baffdec3c3161","Type":"ContainerDied","Data":"a29f994bbb134ca951eaf257756ca42dadf607a9bf0bc8fc0ab7e16550edd250"}
Jul 09 14:40:22.567254 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.567234    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df2893b-e304-464d-a710-8d7094a8e4d2" containerID="cfa53cf5f4fb8f250aee6e6acdcefcefb6e40ee011d30d8be620f6aacfc61c6c" exitCode=0
Jul 09 14:40:22.567336 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.567291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerDied","Data":"cfa53cf5f4fb8f250aee6e6acdcefcefb6e40ee011d30d8be620f6aacfc61c6c"}
Jul 09 14:40:22.568797 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.568771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-psqp4" event={"ID":"886304ba-ee5f-4af4-9435-3a256bc1ede6","Type":"ContainerStarted","Data":"462f77a21609175687c905dca758985752182ea9d0ab740cb47f8a4984838b53"}
Jul 09 14:40:22.574478 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.574435    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vvm8k" podStartSLOduration=2.484918794 podStartE2EDuration="22.574419983s" totalImagesPullingTime="20.089501189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:22.5737971 +0000 UTC m=+23.607325582" watchObservedRunningTime="2026-07-09 14:40:22.574419983 +0000 UTC m=+23.607948465"
Jul 09 14:40:22.589577 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.589541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v22jf" podStartSLOduration=2.497435651 podStartE2EDuration="22.589530177s" totalImagesPullingTime="20.092094526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:22.589388661 +0000 UTC m=+23.622917146" watchObservedRunningTime="2026-07-09 14:40:22.589530177 +0000 UTC m=+23.623058660"
Jul 09 14:40:22.603243 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.603205    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c5htq" podStartSLOduration=2.5692175649999998 podStartE2EDuration="22.603194673s" totalImagesPullingTime="20.033977108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:22.602947168 +0000 UTC m=+23.636475676" watchObservedRunningTime="2026-07-09 14:40:22.603194673 +0000 UTC m=+23.636723156"
Jul 09 14:40:22.634280 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.634228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" podStartSLOduration=21.634215794 podStartE2EDuration="21.634215794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:22.617714398 +0000 UTC m=+23.651242883" watchObservedRunningTime="2026-07-09 14:40:22.634215794 +0000 UTC m=+23.667744275"
Jul 09 14:40:22.649090 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.649050    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2ld92" podStartSLOduration=2.585398734 podStartE2EDuration="22.64903573s" totalImagesPullingTime="20.063636996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:22.633949414 +0000 UTC m=+23.667477896" watchObservedRunningTime="2026-07-09 14:40:22.64903573 +0000 UTC m=+23.682564214"
Jul 09 14:40:22.649231 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.649210    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-psqp4" podStartSLOduration=2.5840282009999997 podStartE2EDuration="22.649198023s" totalImagesPullingTime="20.065169822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:22.649088746 +0000 UTC m=+23.682617228" watchObservedRunningTime="2026-07-09 14:40:22.649198023 +0000 UTC m=+23.682726507"
Jul 09 14:40:22.808425 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:22.808368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:23.357027 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.357001    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 14:40:23.483324 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.483261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:23.483493 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.483266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:23.483493 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:23.483390    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:23.483608 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:23.483494    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:23.573108 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.573075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt" event={"ID":"2a83c9e4-8a68-4de6-8720-e74e360e4610","Type":"ContainerStarted","Data":"5203613df0ce67162a183bfaf1793bf8fe22d885e9bf1d5447724678148a51ea"}
Jul 09 14:40:23.574640 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.574587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xgjt6" event={"ID":"01f91055-f6eb-47b4-8958-65fe57ea065a","Type":"ContainerStarted","Data":"bf1c8fd9a33d624848efa679206a78905a2ad7d9f7f52ac0b9dc28bb8286fe1b"}
Jul 09 14:40:23.576673 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.576642    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"1a76ec413de110e8f73baffdec3c3161","Type":"ContainerStarted","Data":"f1986266b62ee4262dc64fa1fc3a6556a3391be4fe7e8bbf1725c7dce17a071a"}
Jul 09 14:40:23.590860 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.590816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xgjt6" podStartSLOduration=7.691801858 podStartE2EDuration="23.590802564s" totalImagesPullingTime="15.899000706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:23.590263767 +0000 UTC m=+24.623792251" watchObservedRunningTime="2026-07-09 14:40:23.590802564 +0000 UTC m=+24.624331046"
Jul 09 14:40:23.607741 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.607704    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" podStartSLOduration=22.60769098 podStartE2EDuration="22.60769098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:23.606953035 +0000 UTC m=+24.640481518" watchObservedRunningTime="2026-07-09 14:40:23.60769098 +0000 UTC m=+24.641219464"
Jul 09 14:40:23.681041 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.681016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:23.681738 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:23.681721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:24.354704 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.354615    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T14:40:23.357022136Z","UUID":"9662d168-93c3-421c-99cf-4fdc520e6a9c","Handler":null,"Name":"","Endpoint":""}
Jul 09 14:40:24.356392 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.356372    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 14:40:24.356495 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.356416    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 14:40:24.483293 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.483268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:24.483424 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:24.483372    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:24.581602 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.581552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt" event={"ID":"2a83c9e4-8a68-4de6-8720-e74e360e4610","Type":"ContainerStarted","Data":"df77d58311b17a07bc727625c758f99c112fc27ba15511aaf1ac4a299f7f24ac"}
Jul 09 14:40:24.582427 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.582408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c5htq"
Jul 09 14:40:24.600457 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:24.600414    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmlbt" podStartSLOduration=2.392786953 podStartE2EDuration="24.600395551s" totalImagesPullingTime="22.207608598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:24.600099317 +0000 UTC m=+25.633627825" watchObservedRunningTime="2026-07-09 14:40:24.600395551 +0000 UTC m=+25.633924032"
Jul 09 14:40:25.486652 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:25.486622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:25.486837 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:25.486728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:25.486837 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:25.486787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:25.486951 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:25.486891    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:25.587794 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:25.587761    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:40:25.588302 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:25.588269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"41bdbb6551ff93b83af4087f784831097849790bf5a0aadbf767b21df192631f"}
Jul 09 14:40:26.483341 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:26.483312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:26.483521 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:26.483408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:27.483804 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.483562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:27.484459 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.483649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:27.484459 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:27.483875    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:27.484459 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:27.483946    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:27.595617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.595598    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:40:27.596021 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.595999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"3b9bf9df1fab96557c9a3a6e6d53302e01c73acaf4680aa5fc5f0aac6ac85c47"}
Jul 09 14:40:27.596381 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.596272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:27.596381 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.596348    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:27.596501 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.596417    2578 scope.go:117] "RemoveContainer" containerID="b008c2ccb3f3acf93b5061b7ad179dc1921e84195428d539428d2e5108d1be21"
Jul 09 14:40:27.598043 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.597876    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df2893b-e304-464d-a710-8d7094a8e4d2" containerID="c59262ac386ae63e0a653d5ff18a9a895f93c5af6f260038ecb9b57e26553edc" exitCode=0
Jul 09 14:40:27.598128 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.597958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerDied","Data":"c59262ac386ae63e0a653d5ff18a9a895f93c5af6f260038ecb9b57e26553edc"}
Jul 09 14:40:27.613748 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:27.613728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:28.483177 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.483141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:28.483441 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:28.483279    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:28.605092 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.605068    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:40:28.605506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.605479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" event={"ID":"a20a5f44-66dd-4275-8807-af25b9282aaf","Type":"ContainerStarted","Data":"b1d7f38eb5d513e044f32d4ad215c2b79509b558663fa3a17cfcae966a0c4cfd"}
Jul 09 14:40:28.605880 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.605853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:28.622689 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.622670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:40:28.637077 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.637036    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg" podStartSLOduration=8.513422804 podStartE2EDuration="28.637020439s" totalImagesPullingTime="20.123597635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:28.636259838 +0000 UTC m=+29.669788342" watchObservedRunningTime="2026-07-09 14:40:28.637020439 +0000 UTC m=+29.670548921"
Jul 09 14:40:28.853025 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.853002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8jb7"]
Jul 09 14:40:28.853137 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.853127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:28.853244 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:28.853225    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:28.856233 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.856210    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vp7b"]
Jul 09 14:40:28.856341 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.856327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:28.856455 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:28.856435    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:28.867079 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.867054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jnx8p"]
Jul 09 14:40:28.867198 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:28.867150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:28.867258 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:28.867222    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:29.609069 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:29.609033    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df2893b-e304-464d-a710-8d7094a8e4d2" containerID="744d049120be12ba709ea6d25ee9b5dfd7fd28e2b58f73843d93a5bfe4fcc282" exitCode=0
Jul 09 14:40:29.609466 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:29.609080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerDied","Data":"744d049120be12ba709ea6d25ee9b5dfd7fd28e2b58f73843d93a5bfe4fcc282"}
Jul 09 14:40:30.483073 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:30.483036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:30.483073 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:30.483056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:30.483253 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:30.483036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:30.483253 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:30.483159    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:30.483315 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:30.483267    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:30.483358 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:30.483339    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:31.615084 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:31.614881    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df2893b-e304-464d-a710-8d7094a8e4d2" containerID="4f7ad5d2d41163002e4ffcbdbb82a67da51e4805e176598a74809bce64ff6ce6" exitCode=0
Jul 09 14:40:31.615404 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:31.614956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerDied","Data":"4f7ad5d2d41163002e4ffcbdbb82a67da51e4805e176598a74809bce64ff6ce6"}
Jul 09 14:40:32.483606 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:32.483572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:32.483606 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:32.483588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:32.483606 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:32.483573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:32.483833 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:32.483680    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnx8p" podUID="1c01693e-dbb3-4de3-ba5f-a07102d0b303"
Jul 09 14:40:32.483833 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:32.483759    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:40:32.483901 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:32.483878    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8jb7" podUID="c65bcbab-1c89-4bc0-970f-4f17177a5ed1"
Jul 09 14:40:33.060622 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:33.060591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:33.061106 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.060729    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:33.061106 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.060783    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:41:05.060767506 +0000 UTC m=+66.094295978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 14:40:33.160868 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:33.160838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:33.161031 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.161005    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:33.161105 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.161075    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret podName:c65bcbab-1c89-4bc0-970f-4f17177a5ed1 nodeName:}" failed. No retries permitted until 2026-07-09 14:41:05.161055475 +0000 UTC m=+66.194583957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret") pod "global-pull-secret-syncer-w8jb7" (UID: "c65bcbab-1c89-4bc0-970f-4f17177a5ed1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 14:40:33.261874 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:33.261844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:33.262070 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.262043    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 14:40:33.262070 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.262070    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 14:40:33.262283 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.262084    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4kc5k for pod openshift-network-diagnostics/network-check-target-jnx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:33.262283 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:33.262148    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k podName:1c01693e-dbb3-4de3-ba5f-a07102d0b303 nodeName:}" failed. No retries permitted until 2026-07-09 14:41:05.262129484 +0000 UTC m=+66.295657960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4kc5k" (UniqueName: "kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k") pod "network-check-target-jnx8p" (UID: "1c01693e-dbb3-4de3-ba5f-a07102d0b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 14:40:34.275795 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.275771    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeReady"
Jul 09 14:40:34.276087 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.275888    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 14:40:34.310183 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.310160    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7df9b47576-lw8zh"]
Jul 09 14:40:34.313653 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.313628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.314989 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.314950    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"]
Jul 09 14:40:34.316427 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.316408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 14:40:34.316549 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.316532    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hz89z\""
Jul 09 14:40:34.316721 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.316707    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 14:40:34.316778 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.316764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 14:40:34.318732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.318713    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"]
Jul 09 14:40:34.318848 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.318835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:34.322821 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.322759    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"]
Jul 09 14:40:34.323732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.323050    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 14:40:34.323732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.323154    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5p2mz\""
Jul 09 14:40:34.323732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.323155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 14:40:34.323732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.323400    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 14:40:34.323732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.323447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.326558 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.326537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 14:40:34.326643 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.326633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 14:40:34.326931 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.326913    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 14:40:34.327038 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.326922    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 14:40:34.327038 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.326940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 14:40:34.327153 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.327037    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 14:40:34.327472 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.327274    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"]
Jul 09 14:40:34.327472 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.327290    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 14:40:34.327617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.327594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.330560 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.330401    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 14:40:34.330964 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.330944    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-spwkt"]
Jul 09 14:40:34.331063 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.331052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.334451 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.334393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.335858 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.335806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"]
Jul 09 14:40:34.336569 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.336546    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df9b47576-lw8zh"]
Jul 09 14:40:34.338339 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.338291    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 14:40:34.338462 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.338433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-r7h7g\""
Jul 09 14:40:34.339164 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.339029    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 14:40:34.340802 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.339359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"]
Jul 09 14:40:34.340802 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.339399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"]
Jul 09 14:40:34.340802 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.339479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 14:40:34.340802 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.339619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 14:40:34.340802 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.339762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5rhd4\""
Jul 09 14:40:34.343330 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.341505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"]
Jul 09 14:40:34.345907 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.345887    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9z945"]
Jul 09 14:40:34.349600 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.349579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-spwkt"]
Jul 09 14:40:34.349685 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.349678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.352460 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.352433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 14:40:34.352672 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.352655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k7btb\""
Jul 09 14:40:34.352743 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.352674    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 14:40:34.357363 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.357344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9z945"]
Jul 09 14:40:34.472483 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-klusterlet-config\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.472617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-tmp\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.472617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a823838-6515-4fb7-a952-c6626369db38-config-volume\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.472617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2b73b38f-2735-4683-882b-f6fb272afe2a-ca-trust-extracted\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.472617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xck\" (UniqueName: \"kubernetes.io/projected/586d6f50-dfb6-4d37-a8b7-02f370d10abd-kube-api-access-66xck\") pod \"managed-serviceaccount-addon-agent-75f74947b4-zmg75\" (UID: \"586d6f50-dfb6-4d37-a8b7-02f370d10abd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.472617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92162f15-4468-4c81-ad96-0d5fa0967389-cert\") pod \"ingress-canary-spwkt\" (UID: \"92162f15-4468-4c81-ad96-0d5fa0967389\") " pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.472831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-certificates\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.472831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8872040-1139-4109-94fd-a878b137fb4d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:34.472831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9fz\" (UniqueName: \"kubernetes.io/projected/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-kube-api-access-rn9fz\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.472831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.472831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a823838-6515-4fb7-a952-c6626369db38-tmp-dir\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.472831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd746\" (UniqueName: \"kubernetes.io/projected/92162f15-4468-4c81-ad96-0d5fa0967389-kube-api-access-rd746\") pod \"ingress-canary-spwkt\" (UID: \"92162f15-4468-4c81-ad96-0d5fa0967389\") " pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/586d6f50-dfb6-4d37-a8b7-02f370d10abd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f74947b4-zmg75\" (UID: \"586d6f50-dfb6-4d37-a8b7-02f370d10abd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f560d9-dcce-4c73-8eb0-4462800fa525-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-bound-sa-token\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-hub\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8gs\" (UniqueName: \"kubernetes.io/projected/2a823838-6515-4fb7-a952-c6626369db38-kube-api-access-wn8gs\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.472997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.473071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fl9\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-kube-api-access-c4fl9\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.473401 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-trusted-ca\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.473401 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.473401 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-installation-pull-secrets\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.473401 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-image-registry-private-configuration\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.473401 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.473401 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-ca\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.473588 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.473405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rgt\" (UniqueName: \"kubernetes.io/projected/a2f560d9-dcce-4c73-8eb0-4462800fa525-kube-api-access-j8rgt\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.483461 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.483384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:40:34.483581 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.483562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:40:34.483644 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.483576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:40:34.487057 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.487031    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 14:40:34.487147 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.487035    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wgrr9\""
Jul 09 14:40:34.487370 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.487230    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 14:40:34.487370 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.487348    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 14:40:34.487574 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.487348    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 14:40:34.487574 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.487444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d62s8\""
Jul 09 14:40:34.574619 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.574619 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a823838-6515-4fb7-a952-c6626369db38-tmp-dir\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd746\" (UniqueName: \"kubernetes.io/projected/92162f15-4468-4c81-ad96-0d5fa0967389-kube-api-access-rd746\") pod \"ingress-canary-spwkt\" (UID: \"92162f15-4468-4c81-ad96-0d5fa0967389\") " pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/586d6f50-dfb6-4d37-a8b7-02f370d10abd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f74947b4-zmg75\" (UID: \"586d6f50-dfb6-4d37-a8b7-02f370d10abd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f560d9-dcce-4c73-8eb0-4462800fa525-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-bound-sa-token\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-hub\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.574813 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8gs\" (UniqueName: \"kubernetes.io/projected/2a823838-6515-4fb7-a952-c6626369db38-kube-api-access-wn8gs\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fl9\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-kube-api-access-c4fl9\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-trusted-ca\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-installation-pull-secrets\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-image-registry-private-configuration\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a823838-6515-4fb7-a952-c6626369db38-tmp-dir\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575086    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575100    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.574997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.575152 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575154    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:40:35.075134704 +0000 UTC m=+36.108663170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-ca\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575203    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rgt\" (UniqueName: \"kubernetes.io/projected/a2f560d9-dcce-4c73-8eb0-4462800fa525-kube-api-access-j8rgt\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575240    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:40:35.075228817 +0000 UTC m=+36.108757300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-klusterlet-config\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-tmp\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a823838-6515-4fb7-a952-c6626369db38-config-volume\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2b73b38f-2735-4683-882b-f6fb272afe2a-ca-trust-extracted\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xck\" (UniqueName: \"kubernetes.io/projected/586d6f50-dfb6-4d37-a8b7-02f370d10abd-kube-api-access-66xck\") pod \"managed-serviceaccount-addon-agent-75f74947b4-zmg75\" (UID: \"586d6f50-dfb6-4d37-a8b7-02f370d10abd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92162f15-4468-4c81-ad96-0d5fa0967389-cert\") pod \"ingress-canary-spwkt\" (UID: \"92162f15-4468-4c81-ad96-0d5fa0967389\") " pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-certificates\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8872040-1139-4109-94fd-a878b137fb4d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9fz\" (UniqueName: \"kubernetes.io/projected/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-kube-api-access-rn9fz\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.575616 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.575579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f560d9-dcce-4c73-8eb0-4462800fa525-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.576183 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575661    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:40:34.576183 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:34.575707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:35.075691354 +0000 UTC m=+36.109219820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:40:34.577002 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.576386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-trusted-ca\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.577002 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.576918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2b73b38f-2735-4683-882b-f6fb272afe2a-ca-trust-extracted\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.577158 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.577050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a823838-6515-4fb7-a952-c6626369db38-config-volume\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.577435 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.577399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-tmp\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.577699 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.577656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-certificates\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.577902 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.577876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8872040-1139-4109-94fd-a878b137fb4d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-ca\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-image-registry-private-configuration\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/586d6f50-dfb6-4d37-a8b7-02f370d10abd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f74947b4-zmg75\" (UID: \"586d6f50-dfb6-4d37-a8b7-02f370d10abd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-klusterlet-config\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-hub\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.581142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.580841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-installation-pull-secrets\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.581521 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.581425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f560d9-dcce-4c73-8eb0-4462800fa525-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.582019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.582002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92162f15-4468-4c81-ad96-0d5fa0967389-cert\") pod \"ingress-canary-spwkt\" (UID: \"92162f15-4468-4c81-ad96-0d5fa0967389\") " pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.585071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.585023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-bound-sa-token\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.585518 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.585495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9fz\" (UniqueName: \"kubernetes.io/projected/19b955ba-7c89-47aa-a62b-a0bd7f45bbc0-kube-api-access-rn9fz\") pod \"klusterlet-addon-workmgr-857d74c577-bsjz5\" (UID: \"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.586108 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.586085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rgt\" (UniqueName: \"kubernetes.io/projected/a2f560d9-dcce-4c73-8eb0-4462800fa525-kube-api-access-j8rgt\") pod \"cluster-proxy-proxy-agent-6c9ff59bb5-xmznc\" (UID: \"a2f560d9-dcce-4c73-8eb0-4462800fa525\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.586526 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.586505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8gs\" (UniqueName: \"kubernetes.io/projected/2a823838-6515-4fb7-a952-c6626369db38-kube-api-access-wn8gs\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:34.586868 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.586850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xck\" (UniqueName: \"kubernetes.io/projected/586d6f50-dfb6-4d37-a8b7-02f370d10abd-kube-api-access-66xck\") pod \"managed-serviceaccount-addon-agent-75f74947b4-zmg75\" (UID: \"586d6f50-dfb6-4d37-a8b7-02f370d10abd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.587306 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.587287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fl9\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-kube-api-access-c4fl9\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:34.599448 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.599156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd746\" (UniqueName: \"kubernetes.io/projected/92162f15-4468-4c81-ad96-0d5fa0967389-kube-api-access-rd746\") pod \"ingress-canary-spwkt\" (UID: \"92162f15-4468-4c81-ad96-0d5fa0967389\") " pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:34.663093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.663067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:40:34.669888 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.669865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:34.684131 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.684112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:40:34.690063 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:34.690039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-spwkt"
Jul 09 14:40:35.080037 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:35.079999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:35.080213 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:35.080065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:35.080213 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:35.080102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:35.080213 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080160    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:40:35.080213 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080180    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:40:35.080213 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080188    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:40:35.080213 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080158    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:40:35.080515 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080237    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:40:36.080215802 +0000 UTC m=+37.113744286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:40:35.080515 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:36.080247413 +0000 UTC m=+37.113775879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:40:35.080515 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:35.080274    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:40:36.080264497 +0000 UTC m=+37.113792959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:40:36.088624 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.088578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.088646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.088760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.088777    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.088801    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.088872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:40:38.088849611 +0000 UTC m=+39.122378094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.088891    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.088947    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:40:38.088930516 +0000 UTC m=+39.122458993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.089124    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:40:36.089564 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:36.089162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:38.089150222 +0000 UTC m=+39.122678772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:40:36.250550 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.250520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"]
Jul 09 14:40:36.255712 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.255649    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"]
Jul 09 14:40:36.263129 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:36.263101    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b955ba_7c89_47aa_a62b_a0bd7f45bbc0.slice/crio-d205dcf819456be9a98ba68096cca121210258b438b523b5ad076086a92a6a40 WatchSource:0}: Error finding container d205dcf819456be9a98ba68096cca121210258b438b523b5ad076086a92a6a40: Status 404 returned error can't find the container with id d205dcf819456be9a98ba68096cca121210258b438b523b5ad076086a92a6a40
Jul 09 14:40:36.263472 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:36.263447    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod586d6f50_dfb6_4d37_a8b7_02f370d10abd.slice/crio-30f7399537a029711b3ea3d4afde6707e7825ae46bf4b6e9b6ace6f4c50e10a3 WatchSource:0}: Error finding container 30f7399537a029711b3ea3d4afde6707e7825ae46bf4b6e9b6ace6f4c50e10a3: Status 404 returned error can't find the container with id 30f7399537a029711b3ea3d4afde6707e7825ae46bf4b6e9b6ace6f4c50e10a3
Jul 09 14:40:36.266633 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.266610    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-spwkt"]
Jul 09 14:40:36.269943 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:36.269888    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92162f15_4468_4c81_ad96_0d5fa0967389.slice/crio-e5e31688a25f4aa1200679c81ccab258fdcf7d1c4edd1c75e8e0255015610515 WatchSource:0}: Error finding container e5e31688a25f4aa1200679c81ccab258fdcf7d1c4edd1c75e8e0255015610515: Status 404 returned error can't find the container with id e5e31688a25f4aa1200679c81ccab258fdcf7d1c4edd1c75e8e0255015610515
Jul 09 14:40:36.270396 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.270294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"]
Jul 09 14:40:36.272902 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:40:36.272882    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f560d9_dcce_4c73_8eb0_4462800fa525.slice/crio-ced0e04be41432efe4426d41718ba5724d0b58887e5cac50dd2c1ff9c757485c WatchSource:0}: Error finding container ced0e04be41432efe4426d41718ba5724d0b58887e5cac50dd2c1ff9c757485c: Status 404 returned error can't find the container with id ced0e04be41432efe4426d41718ba5724d0b58887e5cac50dd2c1ff9c757485c
Jul 09 14:40:36.626996 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.626892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" event={"ID":"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0","Type":"ContainerStarted","Data":"d205dcf819456be9a98ba68096cca121210258b438b523b5ad076086a92a6a40"}
Jul 09 14:40:36.627961 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.627939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" event={"ID":"a2f560d9-dcce-4c73-8eb0-4462800fa525","Type":"ContainerStarted","Data":"ced0e04be41432efe4426d41718ba5724d0b58887e5cac50dd2c1ff9c757485c"}
Jul 09 14:40:36.628922 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.628898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" event={"ID":"586d6f50-dfb6-4d37-a8b7-02f370d10abd","Type":"ContainerStarted","Data":"30f7399537a029711b3ea3d4afde6707e7825ae46bf4b6e9b6ace6f4c50e10a3"}
Jul 09 14:40:36.631531 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.631510    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df2893b-e304-464d-a710-8d7094a8e4d2" containerID="10255a3484a456f2f90c0cfcb52d5227bd9eab9ef18c38153fcb421dd8343cab" exitCode=0
Jul 09 14:40:36.631602 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.631574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerDied","Data":"10255a3484a456f2f90c0cfcb52d5227bd9eab9ef18c38153fcb421dd8343cab"}
Jul 09 14:40:36.632660 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:36.632639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-spwkt" event={"ID":"92162f15-4468-4c81-ad96-0d5fa0967389","Type":"ContainerStarted","Data":"e5e31688a25f4aa1200679c81ccab258fdcf7d1c4edd1c75e8e0255015610515"}
Jul 09 14:40:37.646166 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:37.646129    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df2893b-e304-464d-a710-8d7094a8e4d2" containerID="2dc589a570cd6777bbf6e48ae6694ed5c176977516bb11b536ef55441baca6c3" exitCode=0
Jul 09 14:40:37.646602 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:37.646214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerDied","Data":"2dc589a570cd6777bbf6e48ae6694ed5c176977516bb11b536ef55441baca6c3"}
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:38.106065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:38.106128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:38.106159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106291    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106350    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:42.106332495 +0000 UTC m=+43.139860961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106496    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106507    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106546    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:40:42.106532679 +0000 UTC m=+43.140061145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106605    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:40:38.106665 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:38.106636    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:40:42.106625566 +0000 UTC m=+43.140154027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:40:38.656373 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:38.656337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" event={"ID":"9df2893b-e304-464d-a710-8d7094a8e4d2","Type":"ContainerStarted","Data":"09e236fdb3a25ace0a88e020ecaf6bb300422d02ab931394d79339bf499ac3bd"}
Jul 09 14:40:38.681996 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:38.681774    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n7r6j" podStartSLOduration=7.225281094 podStartE2EDuration="38.681758077s" totalImagesPullingTime="31.456476983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:38.679635079 +0000 UTC m=+39.713163562" watchObservedRunningTime="2026-07-09 14:40:38.681758077 +0000 UTC m=+39.715286567"
Jul 09 14:40:42.140781 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:42.140599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:42.140781 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.140764    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:42.140825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.140843    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:40:50.140821231 +0000 UTC m=+51.174349711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:42.140910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.140920    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.140933    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.140990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:40:50.140962748 +0000 UTC m=+51.174491208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.141036    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:40:42.141322 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:42.141078    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:40:50.14106372 +0000 UTC m=+51.174592180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:40:45.675829 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.675793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" event={"ID":"a2f560d9-dcce-4c73-8eb0-4462800fa525","Type":"ContainerStarted","Data":"97adbc0dcbf9465040a70b57ca0fe96909d09359bfb928b700b08bb368feb3ae"}
Jul 09 14:40:45.676276 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.675843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" event={"ID":"a2f560d9-dcce-4c73-8eb0-4462800fa525","Type":"ContainerStarted","Data":"07d2de461f97b561bd5e2842fd637347f9cc7d564db6cf3a15b501fcac92e50d"}
Jul 09 14:40:45.676276 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.675867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" event={"ID":"a2f560d9-dcce-4c73-8eb0-4462800fa525","Type":"ContainerStarted","Data":"60cf950894142ee1b3022650a14f50636bed273c2dc1985d4b6d825d1ece7b26"}
Jul 09 14:40:45.677243 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.677222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" event={"ID":"586d6f50-dfb6-4d37-a8b7-02f370d10abd","Type":"ContainerStarted","Data":"bfe8c60fbb17b1ecfa0c40fec3e1d3fc7a3221ada8c54357eaf446c9856e277d"}
Jul 09 14:40:45.678572 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.678552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-spwkt" event={"ID":"92162f15-4468-4c81-ad96-0d5fa0967389","Type":"ContainerStarted","Data":"3955a585ff0491654d93f8ea5561eee4311e8e9203a1f388edbd99851b0a561d"}
Jul 09 14:40:45.679964 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.679938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" event={"ID":"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0","Type":"ContainerStarted","Data":"911dba2fefdc0bceeb83db82ef008160d424a28fc10ccbc90420be01c23515e9"}
Jul 09 14:40:45.696069 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.696030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" podStartSLOduration=19.401568798 podStartE2EDuration="27.69601866s" totalImagesPullingTime="8.294449862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:45.695523231 +0000 UTC m=+46.729051731" watchObservedRunningTime="2026-07-09 14:40:45.69601866 +0000 UTC m=+46.729547143"
Jul 09 14:40:45.713509 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.713469    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" podStartSLOduration=19.409742456 podStartE2EDuration="27.713457923s" totalImagesPullingTime="8.303715467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:45.712988054 +0000 UTC m=+46.746516539" watchObservedRunningTime="2026-07-09 14:40:45.713457923 +0000 UTC m=+46.746986405"
Jul 09 14:40:45.727468 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.727438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" podStartSLOduration=19.416928182 podStartE2EDuration="27.727427019s" totalImagesPullingTime="8.310498837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:45.727162863 +0000 UTC m=+46.760691343" watchObservedRunningTime="2026-07-09 14:40:45.727427019 +0000 UTC m=+46.760955507"
Jul 09 14:40:45.743767 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:45.743735    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-spwkt" podStartSLOduration=3.445673801 podStartE2EDuration="11.743724676s" totalImagesPullingTime="8.298050875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:45.743222781 +0000 UTC m=+46.776751275" watchObservedRunningTime="2026-07-09 14:40:45.743724676 +0000 UTC m=+46.777253157"
Jul 09 14:40:46.685399 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:46.685359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:46.686359 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:46.686342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:40:50.203500 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:50.203468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:50.203511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:40:50.203532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203609    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203611    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203614    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203657    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203674    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:41:06.203657612 +0000 UTC m=+67.237186073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203690    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:41:06.203682411 +0000 UTC m=+67.237210871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:40:50.203868 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:40:50.203708    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:41:06.203701718 +0000 UTC m=+67.237230179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:41:00.624028 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:00.623996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jr7vg"
Jul 09 14:41:05.122525 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.122487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:41:05.125767 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.125750    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 14:41:05.133193 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:05.133172    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 14:41:05.133247 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:05.133229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:42:09.133212228 +0000 UTC m=+130.166740706 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : secret "metrics-daemon-secret" not found
Jul 09 14:41:05.223493 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.223444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:41:05.226376 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.226357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 14:41:05.237582 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.237560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c65bcbab-1c89-4bc0-970f-4f17177a5ed1-original-pull-secret\") pod \"global-pull-secret-syncer-w8jb7\" (UID: \"c65bcbab-1c89-4bc0-970f-4f17177a5ed1\") " pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:41:05.324300 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.324250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:41:05.327339 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.327322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 14:41:05.337177 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.337157    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 14:41:05.347695 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.347673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc5k\" (UniqueName: \"kubernetes.io/projected/1c01693e-dbb3-4de3-ba5f-a07102d0b303-kube-api-access-4kc5k\") pod \"network-check-target-jnx8p\" (UID: \"1c01693e-dbb3-4de3-ba5f-a07102d0b303\") " pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:41:05.403011 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.402923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8jb7"
Jul 09 14:41:05.410471 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.410448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wgrr9\""
Jul 09 14:41:05.417935 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.417906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:41:05.533119 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.533085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8jb7"]
Jul 09 14:41:05.537253 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:41:05.537228    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65bcbab_1c89_4bc0_970f_4f17177a5ed1.slice/crio-422f307698e60b15b7c2e8d0d23198ba7d0628f8be62905a9f216c8f8bb1f9db WatchSource:0}: Error finding container 422f307698e60b15b7c2e8d0d23198ba7d0628f8be62905a9f216c8f8bb1f9db: Status 404 returned error can't find the container with id 422f307698e60b15b7c2e8d0d23198ba7d0628f8be62905a9f216c8f8bb1f9db
Jul 09 14:41:05.548162 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.548143    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jnx8p"]
Jul 09 14:41:05.550844 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:41:05.550824    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c01693e_dbb3_4de3_ba5f_a07102d0b303.slice/crio-6ca71ad809620f4a6899ef91c31c00668db50718a95fd15fd878a88a6de9f79b WatchSource:0}: Error finding container 6ca71ad809620f4a6899ef91c31c00668db50718a95fd15fd878a88a6de9f79b: Status 404 returned error can't find the container with id 6ca71ad809620f4a6899ef91c31c00668db50718a95fd15fd878a88a6de9f79b
Jul 09 14:41:05.737298 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.737259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jnx8p" event={"ID":"1c01693e-dbb3-4de3-ba5f-a07102d0b303","Type":"ContainerStarted","Data":"6ca71ad809620f4a6899ef91c31c00668db50718a95fd15fd878a88a6de9f79b"}
Jul 09 14:41:05.738245 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:05.738208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8jb7" event={"ID":"c65bcbab-1c89-4bc0-970f-4f17177a5ed1","Type":"ContainerStarted","Data":"422f307698e60b15b7c2e8d0d23198ba7d0628f8be62905a9f216c8f8bb1f9db"}
Jul 09 14:41:06.231485 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:06.231446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:06.231502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:06.231584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231614    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231633    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231683    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:41:38.231668538 +0000 UTC m=+99.265196999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231692    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231689    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231742    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:41:38.231726542 +0000 UTC m=+99.265255005 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:41:06.231925 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:06.231791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:41:38.231771142 +0000 UTC m=+99.265299608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:41:10.757276 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:10.757238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jnx8p" event={"ID":"1c01693e-dbb3-4de3-ba5f-a07102d0b303","Type":"ContainerStarted","Data":"0ee9ff235186e895eba17f5340348600143ba669269673ca05d48539e3bb3cb3"}
Jul 09 14:41:10.757726 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:10.757353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:41:10.758797 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:10.758769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8jb7" event={"ID":"c65bcbab-1c89-4bc0-970f-4f17177a5ed1","Type":"ContainerStarted","Data":"9e619bfd3185f11852f99e96d72643427fa73e875648913ceab3de11e895489a"}
Jul 09 14:41:10.776394 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:10.776342    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jnx8p" podStartSLOduration=65.689838074 podStartE2EDuration="1m10.776328376s" totalImagesPullingTime="5.086490302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:41:10.774113517 +0000 UTC m=+71.807642000" watchObservedRunningTime="2026-07-09 14:41:10.776328376 +0000 UTC m=+71.809856836"
Jul 09 14:41:10.789115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:10.789069    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w8jb7" podStartSLOduration=65.683636757 podStartE2EDuration="1m10.789056357s" totalImagesPullingTime="5.1054196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:41:10.788423304 +0000 UTC m=+71.821951808" watchObservedRunningTime="2026-07-09 14:41:10.789056357 +0000 UTC m=+71.822584838"
Jul 09 14:41:38.262205 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:38.262156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:38.262213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:38.262287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262326    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262355    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262395    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262423    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262468    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:42:42.262451252 +0000 UTC m=+163.295979733 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262483    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:42:42.262476899 +0000 UTC m=+163.296005363 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:41:38.262746 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:41:38.262496    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:42:42.262489185 +0000 UTC m=+163.296017646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:41:41.764607 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:41:41.764576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jnx8p"
Jul 09 14:42:09.189397 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:09.189351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:42:09.189883 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:09.189494    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 14:42:09.189883 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:09.189567    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:11.189551474 +0000 UTC m=+252.223079935 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : secret "metrics-daemon-secret" not found
Jul 09 14:42:37.326939 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:37.326896    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a"
Jul 09 14:42:37.335067 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:37.335035    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h" podUID="a8872040-1139-4109-94fd-a878b137fb4d"
Jul 09 14:42:37.397358 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:37.397324    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9z945" podUID="2a823838-6515-4fb7-a952-c6626369db38"
Jul 09 14:42:37.494741 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:37.494713    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:42:38.002268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:38.002244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9z945"
Jul 09 14:42:38.002439 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:38.002244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:42:38.002439 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:38.002247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:42:42.330948 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:42.330914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:42.330962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:42.331013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331081    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331134    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331137    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331156    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df9b47576-lw8zh: secret "image-registry-tls" not found
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert podName:a8872040-1139-4109-94fd-a878b137fb4d nodeName:}" failed. No retries permitted until 2026-07-09 14:44:44.331124744 +0000 UTC m=+285.364653219 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gkb6h" (UID: "a8872040-1139-4109-94fd-a878b137fb4d") : secret "networking-console-plugin-cert" not found
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331222    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls podName:2a823838-6515-4fb7-a952-c6626369db38 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:44.331205184 +0000 UTC m=+285.364733655 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls") pod "dns-default-9z945" (UID: "2a823838-6515-4fb7-a952-c6626369db38") : secret "dns-default-metrics-tls" not found
Jul 09 14:42:42.331422 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:42:42.331234    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls podName:2b73b38f-2735-4683-882b-f6fb272afe2a nodeName:}" failed. No retries permitted until 2026-07-09 14:44:44.331227828 +0000 UTC m=+285.364756289 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls") pod "image-registry-7df9b47576-lw8zh" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a") : secret "image-registry-tls" not found
Jul 09 14:42:45.023168 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:45.023087    2578 generic.go:356] "Generic (PLEG): container finished" podID="19b955ba-7c89-47aa-a62b-a0bd7f45bbc0" containerID="911dba2fefdc0bceeb83db82ef008160d424a28fc10ccbc90420be01c23515e9" exitCode=1
Jul 09 14:42:45.023505 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:45.023162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" event={"ID":"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0","Type":"ContainerDied","Data":"911dba2fefdc0bceeb83db82ef008160d424a28fc10ccbc90420be01c23515e9"}
Jul 09 14:42:45.023505 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:45.023464    2578 scope.go:117] "RemoveContainer" containerID="911dba2fefdc0bceeb83db82ef008160d424a28fc10ccbc90420be01c23515e9"
Jul 09 14:42:45.024637 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:45.024614    2578 generic.go:356] "Generic (PLEG): container finished" podID="586d6f50-dfb6-4d37-a8b7-02f370d10abd" containerID="bfe8c60fbb17b1ecfa0c40fec3e1d3fc7a3221ada8c54357eaf446c9856e277d" exitCode=255
Jul 09 14:42:45.024716 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:45.024689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" event={"ID":"586d6f50-dfb6-4d37-a8b7-02f370d10abd","Type":"ContainerDied","Data":"bfe8c60fbb17b1ecfa0c40fec3e1d3fc7a3221ada8c54357eaf446c9856e277d"}
Jul 09 14:42:45.025020 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:45.025001    2578 scope.go:117] "RemoveContainer" containerID="bfe8c60fbb17b1ecfa0c40fec3e1d3fc7a3221ada8c54357eaf446c9856e277d"
Jul 09 14:42:46.029659 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:46.029618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" event={"ID":"586d6f50-dfb6-4d37-a8b7-02f370d10abd","Type":"ContainerStarted","Data":"17af21cc92b2e446634908ebafbf110fd73089c75a7c2e5412d5101a5fcdccc6"}
Jul 09 14:42:46.031248 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:46.031220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" event={"ID":"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0","Type":"ContainerStarted","Data":"7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864"}
Jul 09 14:42:46.031518 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:46.031501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:42:46.032128 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:46.032112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:42:50.483044 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:42:50.483008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:44:11.212589 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:11.212549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:44:11.213178 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:11.212718    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 14:44:11.213178 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:11.212795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs podName:f26376fc-b839-4b7e-9fa7-160aa9b0faa7 nodeName:}" failed. No retries permitted until 2026-07-09 14:46:13.212773745 +0000 UTC m=+374.246302207 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs") pod "network-metrics-daemon-7vp7b" (UID: "f26376fc-b839-4b7e-9fa7-160aa9b0faa7") : secret "metrics-daemon-secret" not found
Jul 09 14:44:21.816631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.816557    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hclkh"]
Jul 09 14:44:21.819588 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.819566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:21.822271 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.822247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 14:44:21.822396 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.822275    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 14:44:21.822396 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.822317    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jd2rk\""
Jul 09 14:44:21.822491 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.822391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 14:44:21.823769 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.823754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 14:44:21.833274 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.833257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hclkh"]
Jul 09 14:44:21.982747 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.982715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b577cfd6-be9c-449f-8488-87407592f9b1-crio-socket\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:21.982904 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.982775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b577cfd6-be9c-449f-8488-87407592f9b1-data-volume\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:21.982904 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.982822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv64l\" (UniqueName: \"kubernetes.io/projected/b577cfd6-be9c-449f-8488-87407592f9b1-kube-api-access-jv64l\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:21.982904 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.982852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:21.982904 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:21.982888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.083798 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.083723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.083798 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.083797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b577cfd6-be9c-449f-8488-87407592f9b1-crio-socket\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.083995 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.083844    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 14:44:22.083995 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.083853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b577cfd6-be9c-449f-8488-87407592f9b1-data-volume\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.083995 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.083909    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:22.583891397 +0000 UTC m=+263.617419885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : configmap "kube-rbac-proxy" not found
Jul 09 14:44:22.083995 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.083913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b577cfd6-be9c-449f-8488-87407592f9b1-crio-socket\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.083995 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.083926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv64l\" (UniqueName: \"kubernetes.io/projected/b577cfd6-be9c-449f-8488-87407592f9b1-kube-api-access-jv64l\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.083995 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.083972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.084292 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.084090    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:22.084292 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.084143    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:22.584129211 +0000 UTC m=+263.617657676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:22.084292 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.084218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b577cfd6-be9c-449f-8488-87407592f9b1-data-volume\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.092449 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.092425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv64l\" (UniqueName: \"kubernetes.io/projected/b577cfd6-be9c-449f-8488-87407592f9b1-kube-api-access-jv64l\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.588333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.588297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.588333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:22.588341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:22.588530 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.588428    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 14:44:22.588530 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.588436    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:22.588530 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.588476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:23.588463192 +0000 UTC m=+264.621991653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : configmap "kube-rbac-proxy" not found
Jul 09 14:44:22.588530 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:22.588487    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:23.588481696 +0000 UTC m=+264.622010156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:23.594422 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:23.594390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:23.594772 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:23.594429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:23.594772 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:23.594506    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 14:44:23.594772 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:23.594537    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:23.594772 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:23.594556    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:25.594540141 +0000 UTC m=+266.628068615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : configmap "kube-rbac-proxy" not found
Jul 09 14:44:23.594772 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:23.594584    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:25.594572114 +0000 UTC m=+266.628100578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:24.804148 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:24.804121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vvm8k_4f93dcb3-a544-417e-839f-fb1fc4c85494/dns-node-resolver/0.log"
Jul 09 14:44:25.610722 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:25.610682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:25.610722 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:25.610730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:25.610917 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:25.610815    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:25.610917 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:25.610865    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 14:44:25.610917 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:25.610878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:29.610858323 +0000 UTC m=+270.644386801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:25.610917 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:25.610913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:29.610898942 +0000 UTC m=+270.644427403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : configmap "kube-rbac-proxy" not found
Jul 09 14:44:25.805364 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:25.805341    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-psqp4_886304ba-ee5f-4af4-9435-3a256bc1ede6/node-ca/0.log"
Jul 09 14:44:26.605095 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:26.605058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-spwkt_92162f15-4468-4c81-ad96-0d5fa0967389/serve-healthcheck-canary/0.log"
Jul 09 14:44:29.638093 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:29.638063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:29.638451 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:29.638098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:29.638451 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:29.638200    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 14:44:29.638451 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:29.638201    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:29.638451 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:29.638261    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:37.63824344 +0000 UTC m=+278.671772124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:29.638451 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:29.638283    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:37.638273098 +0000 UTC m=+278.671801580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : configmap "kube-rbac-proxy" not found
Jul 09 14:44:34.664793 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:34.664749    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" podUID="a2f560d9-dcce-4c73-8eb0-4462800fa525" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 14:44:37.694058 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:37.694017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:37.694460 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:37.694089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:37.694460 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:37.694168    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:37.694460 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:37.694254    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls podName:b577cfd6-be9c-449f-8488-87407592f9b1 nodeName:}" failed. No retries permitted until 2026-07-09 14:44:53.694233556 +0000 UTC m=+294.727762035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hclkh" (UID: "b577cfd6-be9c-449f-8488-87407592f9b1") : secret "insights-runtime-extractor-tls" not found
Jul 09 14:44:37.694683 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:37.694665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b577cfd6-be9c-449f-8488-87407592f9b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:41.003432 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:41.003382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h" podUID="a8872040-1139-4109-94fd-a878b137fb4d"
Jul 09 14:44:41.003840 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:41.003522    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a"
Jul 09 14:44:41.003840 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:41.003598    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9z945" podUID="2a823838-6515-4fb7-a952-c6626369db38"
Jul 09 14:44:41.364067 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:41.363993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9z945"
Jul 09 14:44:41.364245 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:41.363991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:44:41.364245 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:41.363997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:44:44.342729 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.342679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:44:44.342729 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.342735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:44:44.343397 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.342812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:44:44.345228 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.345202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a823838-6515-4fb7-a952-c6626369db38-metrics-tls\") pod \"dns-default-9z945\" (UID: \"2a823838-6515-4fb7-a952-c6626369db38\") " pod="openshift-dns/dns-default-9z945"
Jul 09 14:44:44.345228 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.345224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"image-registry-7df9b47576-lw8zh\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") " pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:44:44.345379 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.345262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8872040-1139-4109-94fd-a878b137fb4d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gkb6h\" (UID: \"a8872040-1139-4109-94fd-a878b137fb4d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:44:44.369764 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.369735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5p2mz\""
Jul 09 14:44:44.369764 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.369745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k7btb\""
Jul 09 14:44:44.369915 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.369781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hz89z\""
Jul 09 14:44:44.375511 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.375492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"
Jul 09 14:44:44.375590 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.375515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:44:44.375590 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.375531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9z945"
Jul 09 14:44:44.520356 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.520329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df9b47576-lw8zh"]
Jul 09 14:44:44.523839 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:44:44.523816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b73b38f_2735_4683_882b_f6fb272afe2a.slice/crio-4e880a5be00d5bcbdb75e2107cc3fa83baefc136053ad3d98c3dcb3b01e69e60 WatchSource:0}: Error finding container 4e880a5be00d5bcbdb75e2107cc3fa83baefc136053ad3d98c3dcb3b01e69e60: Status 404 returned error can't find the container with id 4e880a5be00d5bcbdb75e2107cc3fa83baefc136053ad3d98c3dcb3b01e69e60
Jul 09 14:44:44.664394 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.664365    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" podUID="a2f560d9-dcce-4c73-8eb0-4462800fa525" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 14:44:44.738767 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.738735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gkb6h"]
Jul 09 14:44:44.742257 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:44.742224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9z945"]
Jul 09 14:44:44.742491 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:44:44.742458    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8872040_1139_4109_94fd_a878b137fb4d.slice/crio-f1cd0f385bbeaae23762cd4c3cdd274c745990ce7475b639473ab2332c6b8b03 WatchSource:0}: Error finding container f1cd0f385bbeaae23762cd4c3cdd274c745990ce7475b639473ab2332c6b8b03: Status 404 returned error can't find the container with id f1cd0f385bbeaae23762cd4c3cdd274c745990ce7475b639473ab2332c6b8b03
Jul 09 14:44:44.745083 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:44:44.745058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a823838_6515_4fb7_a952_c6626369db38.slice/crio-30b37accd944b4e571da390853fae670ea81679ae2df0e7c7a008547030d22ca WatchSource:0}: Error finding container 30b37accd944b4e571da390853fae670ea81679ae2df0e7c7a008547030d22ca: Status 404 returned error can't find the container with id 30b37accd944b4e571da390853fae670ea81679ae2df0e7c7a008547030d22ca
Jul 09 14:44:45.376470 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.376408    2578 generic.go:356] "Generic (PLEG): container finished" podID="19b955ba-7c89-47aa-a62b-a0bd7f45bbc0" containerID="7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864" exitCode=1
Jul 09 14:44:45.376817 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.376482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" event={"ID":"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0","Type":"ContainerDied","Data":"7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864"}
Jul 09 14:44:45.376817 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.376551    2578 scope.go:117] "RemoveContainer" containerID="911dba2fefdc0bceeb83db82ef008160d424a28fc10ccbc90420be01c23515e9"
Jul 09 14:44:45.376902 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.376864    2578 scope.go:117] "RemoveContainer" containerID="7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864"
Jul 09 14:44:45.377098 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:45.377077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-857d74c577-bsjz5_open-cluster-management-agent-addon(19b955ba-7c89-47aa-a62b-a0bd7f45bbc0)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" podUID="19b955ba-7c89-47aa-a62b-a0bd7f45bbc0"
Jul 09 14:44:45.378354 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.378334    2578 generic.go:356] "Generic (PLEG): container finished" podID="586d6f50-dfb6-4d37-a8b7-02f370d10abd" containerID="17af21cc92b2e446634908ebafbf110fd73089c75a7c2e5412d5101a5fcdccc6" exitCode=255
Jul 09 14:44:45.378449 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.378403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" event={"ID":"586d6f50-dfb6-4d37-a8b7-02f370d10abd","Type":"ContainerDied","Data":"17af21cc92b2e446634908ebafbf110fd73089c75a7c2e5412d5101a5fcdccc6"}
Jul 09 14:44:45.378688 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.378672    2578 scope.go:117] "RemoveContainer" containerID="17af21cc92b2e446634908ebafbf110fd73089c75a7c2e5412d5101a5fcdccc6"
Jul 09 14:44:45.378903 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:45.378885    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-75f74947b4-zmg75_open-cluster-management-agent-addon(586d6f50-dfb6-4d37-a8b7-02f370d10abd)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" podUID="586d6f50-dfb6-4d37-a8b7-02f370d10abd"
Jul 09 14:44:45.379894 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.379856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" event={"ID":"2b73b38f-2735-4683-882b-f6fb272afe2a","Type":"ContainerStarted","Data":"bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf"}
Jul 09 14:44:45.380003 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.379904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" event={"ID":"2b73b38f-2735-4683-882b-f6fb272afe2a","Type":"ContainerStarted","Data":"4e880a5be00d5bcbdb75e2107cc3fa83baefc136053ad3d98c3dcb3b01e69e60"}
Jul 09 14:44:45.380003 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.379957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:44:45.381077 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.381044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9z945" event={"ID":"2a823838-6515-4fb7-a952-c6626369db38","Type":"ContainerStarted","Data":"30b37accd944b4e571da390853fae670ea81679ae2df0e7c7a008547030d22ca"}
Jul 09 14:44:45.382066 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.382047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h" event={"ID":"a8872040-1139-4109-94fd-a878b137fb4d","Type":"ContainerStarted","Data":"f1cd0f385bbeaae23762cd4c3cdd274c745990ce7475b639473ab2332c6b8b03"}
Jul 09 14:44:45.386582 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.386497    2578 scope.go:117] "RemoveContainer" containerID="bfe8c60fbb17b1ecfa0c40fec3e1d3fc7a3221ada8c54357eaf446c9856e277d"
Jul 09 14:44:45.431853 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:45.431805    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" podStartSLOduration=285.431786849 podStartE2EDuration="4m45.431786849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:44:45.430801119 +0000 UTC m=+286.464329631" watchObservedRunningTime="2026-07-09 14:44:45.431786849 +0000 UTC m=+286.465315331"
Jul 09 14:44:46.032164 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:46.032117    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:44:46.388123 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:46.388036    2578 scope.go:117] "RemoveContainer" containerID="7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864"
Jul 09 14:44:46.388557 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:46.388229    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-857d74c577-bsjz5_open-cluster-management-agent-addon(19b955ba-7c89-47aa-a62b-a0bd7f45bbc0)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" podUID="19b955ba-7c89-47aa-a62b-a0bd7f45bbc0"
Jul 09 14:44:47.394685 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:47.394651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9z945" event={"ID":"2a823838-6515-4fb7-a952-c6626369db38","Type":"ContainerStarted","Data":"90c0801cab224cde4ddf1c8ce14b32824182237b43f369efb7ab710fb03b4e6b"}
Jul 09 14:44:47.395176 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:47.394712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9z945" event={"ID":"2a823838-6515-4fb7-a952-c6626369db38","Type":"ContainerStarted","Data":"9f38d3272a13921671b555cc1ffd364ae709afa071a36c6bfac6ac6989401989"}
Jul 09 14:44:47.395176 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:47.394764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9z945"
Jul 09 14:44:47.396282 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:47.396260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h" event={"ID":"a8872040-1139-4109-94fd-a878b137fb4d","Type":"ContainerStarted","Data":"d5bdfc673b5a8aa9a7831178b06bb961990dfd6bfd5619828ec6003a306175b4"}
Jul 09 14:44:47.411902 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:47.411855    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9z945" podStartSLOduration=251.610595555 podStartE2EDuration="4m13.41184083s" totalImagesPullingTime="1.801245275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:44:47.411554043 +0000 UTC m=+288.445082521" watchObservedRunningTime="2026-07-09 14:44:47.41184083 +0000 UTC m=+288.445369312"
Jul 09 14:44:47.427642 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:47.427604    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-gkb6h" podStartSLOduration=280.628896306 podStartE2EDuration="4m42.427588451s" totalImagesPullingTime="1.798692145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:44:47.427507363 +0000 UTC m=+288.461035848" watchObservedRunningTime="2026-07-09 14:44:47.427588451 +0000 UTC m=+288.461116936"
Jul 09 14:44:53.483824 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:53.483780    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7vp7b" podUID="f26376fc-b839-4b7e-9fa7-160aa9b0faa7"
Jul 09 14:44:53.710170 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:53.710140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:53.712295 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:53.712276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b577cfd6-be9c-449f-8488-87407592f9b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hclkh\" (UID: \"b577cfd6-be9c-449f-8488-87407592f9b1\") " pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:53.928154 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:53.928127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hclkh"
Jul 09 14:44:54.052614 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.052582    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hclkh"]
Jul 09 14:44:54.055751 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:44:54.055725    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb577cfd6_be9c_449f_8488_87407592f9b1.slice/crio-2248923659f25132df7e02b6dc575d36e7d9c65e3134d2caeeb251615bc33aa1 WatchSource:0}: Error finding container 2248923659f25132df7e02b6dc575d36e7d9c65e3134d2caeeb251615bc33aa1: Status 404 returned error can't find the container with id 2248923659f25132df7e02b6dc575d36e7d9c65e3134d2caeeb251615bc33aa1
Jul 09 14:44:54.418200 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.418167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hclkh" event={"ID":"b577cfd6-be9c-449f-8488-87407592f9b1","Type":"ContainerStarted","Data":"2248923659f25132df7e02b6dc575d36e7d9c65e3134d2caeeb251615bc33aa1"}
Jul 09 14:44:54.664302 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.664272    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" podUID="a2f560d9-dcce-4c73-8eb0-4462800fa525" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 14:44:54.664702 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.664341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc"
Jul 09 14:44:54.664829 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.664800    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"97adbc0dcbf9465040a70b57ca0fe96909d09359bfb928b700b08bb368feb3ae"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 09 14:44:54.664874 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.664849    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" podUID="a2f560d9-dcce-4c73-8eb0-4462800fa525" containerName="service-proxy" containerID="cri-o://97adbc0dcbf9465040a70b57ca0fe96909d09359bfb928b700b08bb368feb3ae" gracePeriod=30
Jul 09 14:44:54.670059 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.670041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:44:54.670351 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.670340    2578 scope.go:117] "RemoveContainer" containerID="7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864"
Jul 09 14:44:54.670480 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:54.670464    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-857d74c577-bsjz5_open-cluster-management-agent-addon(19b955ba-7c89-47aa-a62b-a0bd7f45bbc0)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" podUID="19b955ba-7c89-47aa-a62b-a0bd7f45bbc0"
Jul 09 14:44:54.685124 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.685097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75"
Jul 09 14:44:54.685354 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:54.685337    2578 scope.go:117] "RemoveContainer" containerID="17af21cc92b2e446634908ebafbf110fd73089c75a7c2e5412d5101a5fcdccc6"
Jul 09 14:44:54.685516 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:44:54.685497    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-75f74947b4-zmg75_open-cluster-management-agent-addon(586d6f50-dfb6-4d37-a8b7-02f370d10abd)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" podUID="586d6f50-dfb6-4d37-a8b7-02f370d10abd"
Jul 09 14:44:55.426243 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:55.426206    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2f560d9-dcce-4c73-8eb0-4462800fa525" containerID="97adbc0dcbf9465040a70b57ca0fe96909d09359bfb928b700b08bb368feb3ae" exitCode=2
Jul 09 14:44:55.426424 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:55.426238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" event={"ID":"a2f560d9-dcce-4c73-8eb0-4462800fa525","Type":"ContainerDied","Data":"97adbc0dcbf9465040a70b57ca0fe96909d09359bfb928b700b08bb368feb3ae"}
Jul 09 14:44:55.426424 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:55.426293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9ff59bb5-xmznc" event={"ID":"a2f560d9-dcce-4c73-8eb0-4462800fa525","Type":"ContainerStarted","Data":"3fb3e85d3e9b35d6961be6fc1a420a857bd61aa8b1fddf6312f1b4f6749779bc"}
Jul 09 14:44:57.401683 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:57.401651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9z945"
Jul 09 14:44:58.438369 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:58.438334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hclkh" event={"ID":"b577cfd6-be9c-449f-8488-87407592f9b1","Type":"ContainerStarted","Data":"0015f2fb7c3d2917b9c6ca2d713a488c22b83d88504706a37d4988d2ee35798d"}
Jul 09 14:44:59.301372 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:59.301277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:44:59.301372 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:59.301336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:44:59.305381 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:59.305362    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 14:44:59.448258 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:44:59.448122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hclkh" event={"ID":"b577cfd6-be9c-449f-8488-87407592f9b1","Type":"ContainerStarted","Data":"a525cd99608f61b269a490653aab9757a826895abcce972fb29ccf3a527c8d34"}
Jul 09 14:45:02.458727 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:02.458694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hclkh" event={"ID":"b577cfd6-be9c-449f-8488-87407592f9b1","Type":"ContainerStarted","Data":"11475ed7a13ef974954a4264b5ce475fb380dd252f1fdc8d60a4051e1dcc9ac3"}
Jul 09 14:45:02.479901 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:02.479852    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hclkh" podStartSLOduration=34.22141051 podStartE2EDuration="41.47983678s" totalImagesPullingTime="7.25842627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:44:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:02.478617975 +0000 UTC m=+303.512146494" watchObservedRunningTime="2026-07-09 14:45:02.47983678 +0000 UTC m=+303.513365263"
Jul 09 14:45:04.379534 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:04.379500    2578 patch_prober.go:28] interesting pod/image-registry-7df9b47576-lw8zh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 14:45:04.379887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:04.379552    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 14:45:05.483288 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.483261    2578 scope.go:117] "RemoveContainer" containerID="17af21cc92b2e446634908ebafbf110fd73089c75a7c2e5412d5101a5fcdccc6"
Jul 09 14:45:05.887299 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.887213    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m449j"]
Jul 09 14:45:05.890261 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.890245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:05.893602 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.893574    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 14:45:05.893736 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.893662    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 14:45:05.894884 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.894861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 14:45:05.895017 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.894888    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 14:45:05.895017 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.894893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b4wfv\""
Jul 09 14:45:05.895017 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.894868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 14:45:05.895017 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:05.894939    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 14:45:06.004417 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004511 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-accelerators-collector-config\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004511 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-sys\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004511 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-wtmp\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004628 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-textfile\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004628 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nmz\" (UniqueName: \"kubernetes.io/projected/87831920-1c8e-4f13-8235-05f069e2178a-kube-api-access-86nmz\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004628 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-root\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004715 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87831920-1c8e-4f13-8235-05f069e2178a-metrics-client-ca\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.004715 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.004684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-tls\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105042 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.104966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-accelerators-collector-config\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105157 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-sys\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105157 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-wtmp\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105157 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-textfile\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105157 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nmz\" (UniqueName: \"kubernetes.io/projected/87831920-1c8e-4f13-8235-05f069e2178a-kube-api-access-86nmz\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-root\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-sys\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87831920-1c8e-4f13-8235-05f069e2178a-metrics-client-ca\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-tls\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-root\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-wtmp\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105603 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:45:06.105356    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 14:45:06.105603 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:45:06.105416    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-tls podName:87831920-1c8e-4f13-8235-05f069e2178a nodeName:}" failed. No retries permitted until 2026-07-09 14:45:06.60539918 +0000 UTC m=+307.638927643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-tls") pod "node-exporter-m449j" (UID: "87831920-1c8e-4f13-8235-05f069e2178a") : secret "node-exporter-tls" not found
Jul 09 14:45:06.105603 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-textfile\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105603 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-accelerators-collector-config\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.105728 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.105649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87831920-1c8e-4f13-8235-05f069e2178a-metrics-client-ca\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.107440 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.107423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.115874 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.115852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nmz\" (UniqueName: \"kubernetes.io/projected/87831920-1c8e-4f13-8235-05f069e2178a-kube-api-access-86nmz\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.394157 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.394129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:45:06.471894 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.471869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f74947b4-zmg75" event={"ID":"586d6f50-dfb6-4d37-a8b7-02f370d10abd","Type":"ContainerStarted","Data":"6a2f30f600a2a56c8c24abc87d5f47f06c03c5007ca6acd302d42e5c3abf70f9"}
Jul 09 14:45:06.482925 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.482904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:45:06.483027 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.482945    2578 scope.go:117] "RemoveContainer" containerID="7d4689b7209194f879a185cc152992d719a1f808f0f66d9e945e341d2b3c7864"
Jul 09 14:45:06.610113 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.610082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-tls\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.612381 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.612356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/87831920-1c8e-4f13-8235-05f069e2178a-node-exporter-tls\") pod \"node-exporter-m449j\" (UID: \"87831920-1c8e-4f13-8235-05f069e2178a\") " pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.799128 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.799090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m449j"
Jul 09 14:45:06.810102 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:45:06.810074    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87831920_1c8e_4f13_8235_05f069e2178a.slice/crio-4670b543aee25f93f98ce0dedd20508961748ae907e2a810ab905db6c986c00f WatchSource:0}: Error finding container 4670b543aee25f93f98ce0dedd20508961748ae907e2a810ab905db6c986c00f: Status 404 returned error can't find the container with id 4670b543aee25f93f98ce0dedd20508961748ae907e2a810ab905db6c986c00f
Jul 09 14:45:06.811934 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:06.811915    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 14:45:07.476766 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:07.476733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5" event={"ID":"19b955ba-7c89-47aa-a62b-a0bd7f45bbc0","Type":"ContainerStarted","Data":"8d5379221395875f482998965d899087ac2135a68c8a7ec7e1d6283a15553c47"}
Jul 09 14:45:07.477087 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:07.477036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:45:07.477932 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:07.477911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m449j" event={"ID":"87831920-1c8e-4f13-8235-05f069e2178a","Type":"ContainerStarted","Data":"4670b543aee25f93f98ce0dedd20508961748ae907e2a810ab905db6c986c00f"}
Jul 09 14:45:07.478634 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:07.478614    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857d74c577-bsjz5"
Jul 09 14:45:08.482290 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:08.482252    2578 generic.go:356] "Generic (PLEG): container finished" podID="87831920-1c8e-4f13-8235-05f069e2178a" containerID="2d578bc7043b44b1c7c2a1f3e44948b6a8e0d9363eec95f7e5510307eb3c84ca" exitCode=0
Jul 09 14:45:08.482655 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:08.482335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m449j" event={"ID":"87831920-1c8e-4f13-8235-05f069e2178a","Type":"ContainerDied","Data":"2d578bc7043b44b1c7c2a1f3e44948b6a8e0d9363eec95f7e5510307eb3c84ca"}
Jul 09 14:45:09.487510 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:09.487478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m449j" event={"ID":"87831920-1c8e-4f13-8235-05f069e2178a","Type":"ContainerStarted","Data":"5e4019ce36c7f9970cb2228edba8769ab711f219b95f778067fe130769cf81a6"}
Jul 09 14:45:09.487510 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:09.487518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m449j" event={"ID":"87831920-1c8e-4f13-8235-05f069e2178a","Type":"ContainerStarted","Data":"5e77816a4ba2b09a15467c0c2da3baf8ab42d847a047e40225825c127f0ccc1a"}
Jul 09 14:45:09.510439 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:09.510394    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m449j" podStartSLOduration=3.425631701 podStartE2EDuration="4.510381351s" totalImagesPullingTime="1.023834861s" totalInitContainerRuntime="60.914789ms" isStatefulPod=true podCreationTimestamp="2026-07-09 14:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:09.509097194 +0000 UTC m=+310.542625694" watchObservedRunningTime="2026-07-09 14:45:09.510381351 +0000 UTC m=+310.543909833"
Jul 09 14:45:10.157169 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:10.157138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df9b47576-lw8zh"]
Jul 09 14:45:35.181605 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.181548    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a" containerName="registry" containerID="cri-o://bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf" gracePeriod=30
Jul 09 14:45:35.409713 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.409690    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:45:35.506132 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4fl9\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-kube-api-access-c4fl9\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506132 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-image-registry-private-configuration\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2b73b38f-2735-4683-882b-f6fb272afe2a-ca-trust-extracted\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-certificates\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-trusted-ca\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-installation-pull-secrets\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-bound-sa-token\") pod \"2b73b38f-2735-4683-882b-f6fb272afe2a\" (UID: \"2b73b38f-2735-4683-882b-f6fb272afe2a\") "
Jul 09 14:45:35.506800 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506725    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 14:45:35.506900 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.506804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 14:45:35.508534 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.508494    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:45:35.508649 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.508619    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-kube-api-access-c4fl9" (OuterVolumeSpecName: "kube-api-access-c4fl9") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "kube-api-access-c4fl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:45:35.508837 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.508719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:45:35.508931 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.508840    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:45:35.508931 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.508783    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:45:35.515203 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.515178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b73b38f-2735-4683-882b-f6fb272afe2a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2b73b38f-2735-4683-882b-f6fb272afe2a" (UID: "2b73b38f-2735-4683-882b-f6fb272afe2a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 14:45:35.572626 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.572600    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b73b38f-2735-4683-882b-f6fb272afe2a" containerID="bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf" exitCode=0
Jul 09 14:45:35.572718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.572637    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh"
Jul 09 14:45:35.572718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.572638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" event={"ID":"2b73b38f-2735-4683-882b-f6fb272afe2a","Type":"ContainerDied","Data":"bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf"}
Jul 09 14:45:35.572718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.572682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df9b47576-lw8zh" event={"ID":"2b73b38f-2735-4683-882b-f6fb272afe2a","Type":"ContainerDied","Data":"4e880a5be00d5bcbdb75e2107cc3fa83baefc136053ad3d98c3dcb3b01e69e60"}
Jul 09 14:45:35.572718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.572708    2578 scope.go:117] "RemoveContainer" containerID="bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf"
Jul 09 14:45:35.580993 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.580961    2578 scope.go:117] "RemoveContainer" containerID="bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf"
Jul 09 14:45:35.581269 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:45:35.581249    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf\": container with ID starting with bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf not found: ID does not exist" containerID="bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf"
Jul 09 14:45:35.581323 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.581279    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf"} err="failed to get container status \"bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf\": rpc error: code = NotFound desc = could not find container \"bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf\": container with ID starting with bc01d3d572f68862385b81e4daa9eae7785e4bd20bf5279e8e2f255f11a260bf not found: ID does not exist"
Jul 09 14:45:35.593706 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.593684    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df9b47576-lw8zh"]
Jul 09 14:45:35.598643 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.598625    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7df9b47576-lw8zh"]
Jul 09 14:45:35.607750 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607726    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4fl9\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-kube-api-access-c4fl9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607750 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607750    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-image-registry-private-configuration\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607761    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2b73b38f-2735-4683-882b-f6fb272afe2a-ca-trust-extracted\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607772    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-certificates\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607781    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-registry-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607789    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b73b38f-2735-4683-882b-f6fb272afe2a-trusted-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607798    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2b73b38f-2735-4683-882b-f6fb272afe2a-installation-pull-secrets\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:35.607910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:35.607811    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b73b38f-2735-4683-882b-f6fb272afe2a-bound-sa-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:45:37.487177 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:45:37.487147    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a" path="/var/lib/kubelet/pods/2b73b38f-2735-4683-882b-f6fb272afe2a/volumes"
Jul 09 14:46:13.258926 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:13.258890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:46:13.261136 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:13.261114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26376fc-b839-4b7e-9fa7-160aa9b0faa7-metrics-certs\") pod \"network-metrics-daemon-7vp7b\" (UID: \"f26376fc-b839-4b7e-9fa7-160aa9b0faa7\") " pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:46:13.387053 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:13.387030    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d62s8\""
Jul 09 14:46:13.394325 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:13.394307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vp7b"
Jul 09 14:46:13.516060 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:13.516001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vp7b"]
Jul 09 14:46:13.519159 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:46:13.519128    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26376fc_b839_4b7e_9fa7_160aa9b0faa7.slice/crio-66264b7fc859e22cea3cc7f6d19eb8bd56255e0d053c15dc02ef58aa07e0c698 WatchSource:0}: Error finding container 66264b7fc859e22cea3cc7f6d19eb8bd56255e0d053c15dc02ef58aa07e0c698: Status 404 returned error can't find the container with id 66264b7fc859e22cea3cc7f6d19eb8bd56255e0d053c15dc02ef58aa07e0c698
Jul 09 14:46:13.693287 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:13.693254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vp7b" event={"ID":"f26376fc-b839-4b7e-9fa7-160aa9b0faa7","Type":"ContainerStarted","Data":"66264b7fc859e22cea3cc7f6d19eb8bd56255e0d053c15dc02ef58aa07e0c698"}
Jul 09 14:46:14.698019 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:14.697995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vp7b" event={"ID":"f26376fc-b839-4b7e-9fa7-160aa9b0faa7","Type":"ContainerStarted","Data":"ee7608c2496660fba03f5efd2def1e40c0756036bb9a731b0901aec6b547f517"}
Jul 09 14:46:15.703117 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:15.703082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vp7b" event={"ID":"f26376fc-b839-4b7e-9fa7-160aa9b0faa7","Type":"ContainerStarted","Data":"644b2dad24c7b631372239da4f2541e3b1e873bcc9687689ac0f7e88ca21762e"}
Jul 09 14:46:15.720658 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:46:15.720614    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7vp7b" podStartSLOduration=374.684759456 podStartE2EDuration="6m15.720600007s" totalImagesPullingTime="1.035840551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:46:15.719922622 +0000 UTC m=+376.753451108" watchObservedRunningTime="2026-07-09 14:46:15.720600007 +0000 UTC m=+376.754128528"
Jul 09 14:48:05.229433 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.229401    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"]
Jul 09 14:48:05.229887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.229616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a" containerName="registry"
Jul 09 14:48:05.229887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.229626    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a" containerName="registry"
Jul 09 14:48:05.229887 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.229669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b73b38f-2735-4683-882b-f6fb272afe2a" containerName="registry"
Jul 09 14:48:05.232336 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.232321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.235268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.235249    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 14:48:05.235387 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.235250    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 14:48:05.236740 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.236715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-stdrf\""
Jul 09 14:48:05.240758 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.240741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"]
Jul 09 14:48:05.372171 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.372149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxsr\" (UniqueName: \"kubernetes.io/projected/e1fc50a3-2928-478c-92d7-98d7715bd35e-kube-api-access-ddxsr\") pod \"cert-manager-cainjector-559cdd8c87-8fqvm\" (UID: \"e1fc50a3-2928-478c-92d7-98d7715bd35e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.372264 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.372186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1fc50a3-2928-478c-92d7-98d7715bd35e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8fqvm\" (UID: \"e1fc50a3-2928-478c-92d7-98d7715bd35e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.472811 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.472789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxsr\" (UniqueName: \"kubernetes.io/projected/e1fc50a3-2928-478c-92d7-98d7715bd35e-kube-api-access-ddxsr\") pod \"cert-manager-cainjector-559cdd8c87-8fqvm\" (UID: \"e1fc50a3-2928-478c-92d7-98d7715bd35e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.472880 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.472825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1fc50a3-2928-478c-92d7-98d7715bd35e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8fqvm\" (UID: \"e1fc50a3-2928-478c-92d7-98d7715bd35e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.481313 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.481252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1fc50a3-2928-478c-92d7-98d7715bd35e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8fqvm\" (UID: \"e1fc50a3-2928-478c-92d7-98d7715bd35e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.481399 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.481360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxsr\" (UniqueName: \"kubernetes.io/projected/e1fc50a3-2928-478c-92d7-98d7715bd35e-kube-api-access-ddxsr\") pod \"cert-manager-cainjector-559cdd8c87-8fqvm\" (UID: \"e1fc50a3-2928-478c-92d7-98d7715bd35e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.541587 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.541568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"
Jul 09 14:48:05.660216 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:05.660189    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm"]
Jul 09 14:48:05.663219 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:48:05.663192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1fc50a3_2928_478c_92d7_98d7715bd35e.slice/crio-ed31cbdab2239a45b95345030c12670687c2809360ebb5c7797b2369f0bcc90a WatchSource:0}: Error finding container ed31cbdab2239a45b95345030c12670687c2809360ebb5c7797b2369f0bcc90a: Status 404 returned error can't find the container with id ed31cbdab2239a45b95345030c12670687c2809360ebb5c7797b2369f0bcc90a
Jul 09 14:48:06.052390 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:06.052362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm" event={"ID":"e1fc50a3-2928-478c-92d7-98d7715bd35e","Type":"ContainerStarted","Data":"ed31cbdab2239a45b95345030c12670687c2809360ebb5c7797b2369f0bcc90a"}
Jul 09 14:48:09.065847 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:09.065812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm" event={"ID":"e1fc50a3-2928-478c-92d7-98d7715bd35e","Type":"ContainerStarted","Data":"a33f1bc122ffd7e86fc79b80c09999c04e313322da5ff6a9db9653489e28aac8"}
Jul 09 14:48:09.083620 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:09.083573    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8fqvm" podStartSLOduration=1.121476992 podStartE2EDuration="4.083559905s" totalImagesPullingTime="2.962082913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:48:09.082703821 +0000 UTC m=+490.116232298" watchObservedRunningTime="2026-07-09 14:48:09.083559905 +0000 UTC m=+490.117088386"
Jul 09 14:48:34.845428 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.845394    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"]
Jul 09 14:48:34.849575 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.849555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:34.853014 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.852990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 14:48:34.853236 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.853218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9mjvj\""
Jul 09 14:48:34.853336 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.853288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 14:48:34.853433 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.853413    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 14:48:34.853470 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.853451    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 14:48:34.868669 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.868650    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"]
Jul 09 14:48:34.965791 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.965765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlq6\" (UniqueName: \"kubernetes.io/projected/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-kube-api-access-4rlq6\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:34.965912 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.965798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:34.965912 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.965817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-manifests\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:34.965912 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.965874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:34.965912 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:34.965910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-tmp\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.067080 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.067055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlq6\" (UniqueName: \"kubernetes.io/projected/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-kube-api-access-4rlq6\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.067176 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.067089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.067176 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.067109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-manifests\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.067176 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.067135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.067176 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.067163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-tmp\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.067544 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.067527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-manifests\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.069452 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.069434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-tmp\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.069566 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.069549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.069745 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.069727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.083329 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.083308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlq6\" (UniqueName: \"kubernetes.io/projected/ff787fdf-c6f0-4676-9d4b-a197f7ba2346-kube-api-access-4rlq6\") pod \"opendatahub-operator-controller-manager-6dd54665dd-qjncx\" (UID: \"ff787fdf-c6f0-4676-9d4b-a197f7ba2346\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.158818 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.158790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:35.291204 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:35.291182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"]
Jul 09 14:48:35.295229 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:48:35.295186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff787fdf_c6f0_4676_9d4b_a197f7ba2346.slice/crio-15d31c3613848a7121f8df828e49a448cb1951efec5c840990584c074c3dc47f WatchSource:0}: Error finding container 15d31c3613848a7121f8df828e49a448cb1951efec5c840990584c074c3dc47f: Status 404 returned error can't find the container with id 15d31c3613848a7121f8df828e49a448cb1951efec5c840990584c074c3dc47f
Jul 09 14:48:36.160761 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:36.160689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx" event={"ID":"ff787fdf-c6f0-4676-9d4b-a197f7ba2346","Type":"ContainerStarted","Data":"15d31c3613848a7121f8df828e49a448cb1951efec5c840990584c074c3dc47f"}
Jul 09 14:48:39.173960 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:39.173922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx" event={"ID":"ff787fdf-c6f0-4676-9d4b-a197f7ba2346","Type":"ContainerStarted","Data":"aae4666a72df5cb3b6ce15f796a8541dbb177a14289d05491e252e03116af03c"}
Jul 09 14:48:40.178872 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:40.178834    2578 generic.go:356] "Generic (PLEG): container finished" podID="ff787fdf-c6f0-4676-9d4b-a197f7ba2346" containerID="aae4666a72df5cb3b6ce15f796a8541dbb177a14289d05491e252e03116af03c" exitCode=0
Jul 09 14:48:40.179259 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:40.178918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx" event={"ID":"ff787fdf-c6f0-4676-9d4b-a197f7ba2346","Type":"ContainerDied","Data":"aae4666a72df5cb3b6ce15f796a8541dbb177a14289d05491e252e03116af03c"}
Jul 09 14:48:41.184356 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:41.184319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx" event={"ID":"ff787fdf-c6f0-4676-9d4b-a197f7ba2346","Type":"ContainerStarted","Data":"bf34b6eb32b351431a2ccb6db142fd84d66bb9541649e9d5e06d339ccfa786cc"}
Jul 09 14:48:41.184741 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:41.184441    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:41.210851 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:41.210801    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx" podStartSLOduration=2.794507513 podStartE2EDuration="7.210783445s" totalImagesPullingTime="3.393702388s" totalInitContainerRuntime="1.022573544s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:48:41.208703535 +0000 UTC m=+522.242232019" watchObservedRunningTime="2026-07-09 14:48:41.210783445 +0000 UTC m=+522.244312002"
Jul 09 14:48:52.189072 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:52.188997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dd54665dd-qjncx"
Jul 09 14:48:54.537376 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.537343    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mks58"]
Jul 09 14:48:54.540333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.540318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:54.545869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.545848    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 14:48:54.545869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.545861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8g4mh\""
Jul 09 14:48:54.560025 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.560002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mks58"]
Jul 09 14:48:54.598538 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.598506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8adf929e-0714-47e3-b4ee-bf506720931a-cert\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:54.598670 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.598544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnkb\" (UniqueName: \"kubernetes.io/projected/8adf929e-0714-47e3-b4ee-bf506720931a-kube-api-access-7fnkb\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:54.699518 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.699481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8adf929e-0714-47e3-b4ee-bf506720931a-cert\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:54.699632 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.699526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnkb\" (UniqueName: \"kubernetes.io/projected/8adf929e-0714-47e3-b4ee-bf506720931a-kube-api-access-7fnkb\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:54.699632 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:48:54.699616    2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 14:48:54.699709 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:48:54.699679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8adf929e-0714-47e3-b4ee-bf506720931a-cert podName:8adf929e-0714-47e3-b4ee-bf506720931a nodeName:}" failed. No retries permitted until 2026-07-09 14:48:55.199661547 +0000 UTC m=+536.233190007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8adf929e-0714-47e3-b4ee-bf506720931a-cert") pod "odh-model-controller-7998c67dc4-mks58" (UID: "8adf929e-0714-47e3-b4ee-bf506720931a") : secret "odh-model-controller-webhook-cert" not found
Jul 09 14:48:54.713664 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:54.713641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnkb\" (UniqueName: \"kubernetes.io/projected/8adf929e-0714-47e3-b4ee-bf506720931a-kube-api-access-7fnkb\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:55.203777 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.203745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8adf929e-0714-47e3-b4ee-bf506720931a-cert\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:55.206018 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.206000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8adf929e-0714-47e3-b4ee-bf506720931a-cert\") pod \"odh-model-controller-7998c67dc4-mks58\" (UID: \"8adf929e-0714-47e3-b4ee-bf506720931a\") " pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:55.283574 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.283541    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"]
Jul 09 14:48:55.286627 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.286612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.289877 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.289857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 14:48:55.290120 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.290104    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 14:48:55.290182 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.290109    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 14:48:55.290182 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.290109    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 14:48:55.303444 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.303417    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8mht6"]
Jul 09 14:48:55.304658 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.304632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae374918-32df-4b1d-859d-0f74b3ca398b-tls-certs\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.304836 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.304817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae374918-32df-4b1d-859d-0f74b3ca398b-tmp\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.305128 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.304937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4vq\" (UniqueName: \"kubernetes.io/projected/ae374918-32df-4b1d-859d-0f74b3ca398b-kube-api-access-ns4vq\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.307879 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.307853    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"]
Jul 09 14:48:55.307993 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.307963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:55.312185 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.312166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 09 14:48:55.322268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.322238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8mht6"]
Jul 09 14:48:55.406390 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.406367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae374918-32df-4b1d-859d-0f74b3ca398b-tmp\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.406488 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.406396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4vq\" (UniqueName: \"kubernetes.io/projected/ae374918-32df-4b1d-859d-0f74b3ca398b-kube-api-access-ns4vq\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.406488 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.406419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pnxf\" (UniqueName: \"kubernetes.io/projected/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-kube-api-access-7pnxf\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:55.406593 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.406560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-tls-certs\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:55.406648 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.406617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae374918-32df-4b1d-859d-0f74b3ca398b-tls-certs\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.408545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.408518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae374918-32df-4b1d-859d-0f74b3ca398b-tmp\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.408803 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.408785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae374918-32df-4b1d-859d-0f74b3ca398b-tls-certs\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.414702 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.414680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4vq\" (UniqueName: \"kubernetes.io/projected/ae374918-32df-4b1d-859d-0f74b3ca398b-kube-api-access-ns4vq\") pod \"kube-auth-proxy-7c698c67c6-9qkln\" (UID: \"ae374918-32df-4b1d-859d-0f74b3ca398b\") " pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.450446 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.450423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:48:55.507801 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.507770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-tls-certs\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:55.507936 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.507881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pnxf\" (UniqueName: \"kubernetes.io/projected/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-kube-api-access-7pnxf\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:55.508012 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:48:55.507934    2578 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 09 14:48:55.508254 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:48:55.508110    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-tls-certs podName:02e1b7c0-7f72-4dd3-89c1-b8e537788d75 nodeName:}" failed. No retries permitted until 2026-07-09 14:48:56.008087344 +0000 UTC m=+537.041615876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-tls-certs") pod "model-serving-api-54d89bd5fc-8mht6" (UID: "02e1b7c0-7f72-4dd3-89c1-b8e537788d75") : secret "model-serving-api-tls" not found
Jul 09 14:48:55.519834 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.519810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pnxf\" (UniqueName: \"kubernetes.io/projected/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-kube-api-access-7pnxf\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:55.574121 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.574019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mks58"]
Jul 09 14:48:55.576501 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:48:55.576473    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adf929e_0714_47e3_b4ee_bf506720931a.slice/crio-a7c73791169c1c7f8714df03fd6977b1013161e859b284c2d71e03c0767de46d WatchSource:0}: Error finding container a7c73791169c1c7f8714df03fd6977b1013161e859b284c2d71e03c0767de46d: Status 404 returned error can't find the container with id a7c73791169c1c7f8714df03fd6977b1013161e859b284c2d71e03c0767de46d
Jul 09 14:48:55.595861 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.595838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"
Jul 09 14:48:55.724268 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:55.724246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln"]
Jul 09 14:48:55.726398 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:48:55.726376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae374918_32df_4b1d_859d_0f74b3ca398b.slice/crio-916837899b41e884aeafd04d3c3e0f922552d483a8265754cefbfe64127d1914 WatchSource:0}: Error finding container 916837899b41e884aeafd04d3c3e0f922552d483a8265754cefbfe64127d1914: Status 404 returned error can't find the container with id 916837899b41e884aeafd04d3c3e0f922552d483a8265754cefbfe64127d1914
Jul 09 14:48:56.011994 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.011897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-tls-certs\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:56.014311 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.014286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e1b7c0-7f72-4dd3-89c1-b8e537788d75-tls-certs\") pod \"model-serving-api-54d89bd5fc-8mht6\" (UID: \"02e1b7c0-7f72-4dd3-89c1-b8e537788d75\") " pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:56.219510 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.219469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:48:56.237566 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.237537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln" event={"ID":"ae374918-32df-4b1d-859d-0f74b3ca398b","Type":"ContainerStarted","Data":"916837899b41e884aeafd04d3c3e0f922552d483a8265754cefbfe64127d1914"}
Jul 09 14:48:56.238737 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.238713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mks58" event={"ID":"8adf929e-0714-47e3-b4ee-bf506720931a","Type":"ContainerStarted","Data":"a7c73791169c1c7f8714df03fd6977b1013161e859b284c2d71e03c0767de46d"}
Jul 09 14:48:56.364570 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.364536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8mht6"]
Jul 09 14:48:56.367665 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:48:56.367633    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e1b7c0_7f72_4dd3_89c1_b8e537788d75.slice/crio-89fe60e34ff160195403eadc9ad8e5cf09b0ca0763b85cbe3348ed5dbefcff1d WatchSource:0}: Error finding container 89fe60e34ff160195403eadc9ad8e5cf09b0ca0763b85cbe3348ed5dbefcff1d: Status 404 returned error can't find the container with id 89fe60e34ff160195403eadc9ad8e5cf09b0ca0763b85cbe3348ed5dbefcff1d
Jul 09 14:48:56.439018 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.438988    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"]
Jul 09 14:48:56.442182 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.442164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.445473 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.445434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 14:48:56.445581 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.445472    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 14:48:56.445581 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.445472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 14:48:56.445581 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.445517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s55xz\""
Jul 09 14:48:56.445581 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.445434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 14:48:56.455653 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.455633    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"]
Jul 09 14:48:56.517030 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.516938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.517030 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.517005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-token\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.517246 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.517033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-cacerts\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.517246 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.517078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.517246 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.517119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.517246 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.517148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e275fb8-ee11-4d55-9d35-86dd661ede7a-local-certs\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.517246 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.517191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cbz\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-kube-api-access-j6cbz\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618323 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618323 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-token\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618323 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-cacerts\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618878 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618878 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618878 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e275fb8-ee11-4d55-9d35-86dd661ede7a-local-certs\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.618878 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.618818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cbz\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-kube-api-access-j6cbz\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.619369 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.619347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.621241 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.621219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e275fb8-ee11-4d55-9d35-86dd661ede7a-local-certs\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.621351 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.621297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.621429 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.621404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-cacerts\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.621548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.621412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.626475 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.626438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-token\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.626732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.626714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cbz\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-kube-api-access-j6cbz\") pod \"istiod-openshift-gateway-7574459656-5m88v\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.754383 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.754104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:48:56.913287 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:56.913214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"]
Jul 09 14:48:56.918483 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:48:56.918444    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e275fb8_ee11_4d55_9d35_86dd661ede7a.slice/crio-d5b6c93c451552fd477ea010187f3c30879c5ec2dcfcc9d0c355255cc04e8964 WatchSource:0}: Error finding container d5b6c93c451552fd477ea010187f3c30879c5ec2dcfcc9d0c355255cc04e8964: Status 404 returned error can't find the container with id d5b6c93c451552fd477ea010187f3c30879c5ec2dcfcc9d0c355255cc04e8964
Jul 09 14:48:57.245346 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:57.245249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v" event={"ID":"8e275fb8-ee11-4d55-9d35-86dd661ede7a","Type":"ContainerStarted","Data":"d5b6c93c451552fd477ea010187f3c30879c5ec2dcfcc9d0c355255cc04e8964"}
Jul 09 14:48:57.246869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:57.246826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6" event={"ID":"02e1b7c0-7f72-4dd3-89c1-b8e537788d75","Type":"ContainerStarted","Data":"89fe60e34ff160195403eadc9ad8e5cf09b0ca0763b85cbe3348ed5dbefcff1d"}
Jul 09 14:48:59.770971 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:48:59.770947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 14:49:00.262090 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.262017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln" event={"ID":"ae374918-32df-4b1d-859d-0f74b3ca398b","Type":"ContainerStarted","Data":"b8aa543a000eb1308f8aa59b318110529a5a7c1c2546b90735962ee46fa3032b"}
Jul 09 14:49:00.263816 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.263785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mks58" event={"ID":"8adf929e-0714-47e3-b4ee-bf506720931a","Type":"ContainerStarted","Data":"19ee3b053284268281ae26c90e2e7709f99f4565e94f123d195251fd8fac6194"}
Jul 09 14:49:00.263952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.263886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:49:00.265646 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.265623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6" event={"ID":"02e1b7c0-7f72-4dd3-89c1-b8e537788d75","Type":"ContainerStarted","Data":"84ac06b54b415d6e3ac291b9caad4ee819aea26f8ecabade23fd8a6c2817281b"}
Jul 09 14:49:00.265765 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.265685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:49:00.280237 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.280194    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c698c67c6-9qkln" podStartSLOduration=1.240783777 podStartE2EDuration="5.280176017s" totalImagesPullingTime="4.03939224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:00.279670088 +0000 UTC m=+541.313198574" watchObservedRunningTime="2026-07-09 14:49:00.280176017 +0000 UTC m=+541.313704500"
Jul 09 14:49:00.321880 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.320429    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6" podStartSLOduration=1.9320399560000001 podStartE2EDuration="5.320393787s" totalImagesPullingTime="3.388353831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:00.298734919 +0000 UTC m=+541.332263402" watchObservedRunningTime="2026-07-09 14:49:00.320393787 +0000 UTC m=+541.353922268"
Jul 09 14:49:00.321880 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:00.320896    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mks58" podStartSLOduration=2.128531316 podStartE2EDuration="6.320883841s" totalImagesPullingTime="4.192352525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:00.320555835 +0000 UTC m=+541.354084310" watchObservedRunningTime="2026-07-09 14:49:00.320883841 +0000 UTC m=+541.354412325"
Jul 09 14:49:01.022016 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:01.021966    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 14:49:01.022416 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:01.022054    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 14:49:01.270830 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:01.270699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v" event={"ID":"8e275fb8-ee11-4d55-9d35-86dd661ede7a","Type":"ContainerStarted","Data":"060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a"}
Jul 09 14:49:01.271150 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:01.271124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:49:01.297317 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:01.297270    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v" podStartSLOduration=1.197334038 podStartE2EDuration="5.297255479s" totalImagesPullingTime="4.099921441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:01.29591602 +0000 UTC m=+542.329444522" watchObservedRunningTime="2026-07-09 14:49:01.297255479 +0000 UTC m=+542.330783971"
Jul 09 14:49:02.279527 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:02.279499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:49:04.934145 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.934111    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"]
Jul 09 14:49:04.937525 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.937505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:04.937901 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.937877    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"]
Jul 09 14:49:04.941209 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.941190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:04.941419 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.941400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 09 14:49:04.941536 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.941422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bbqxk\""
Jul 09 14:49:04.945384 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.945368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 09 14:49:04.945663 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.945639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lkk2g\""
Jul 09 14:49:04.951189 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.951167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"]
Jul 09 14:49:04.961718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.961695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"]
Jul 09 14:49:04.985234 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.985210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62c147fe-9e8b-4025-b734-45da4ed056a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:04.985320 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.985240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c151bf-55cf-47a5-9b0c-c550b60f8633-cert\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:04.985320 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.985264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlfb\" (UniqueName: \"kubernetes.io/projected/62c147fe-9e8b-4025-b734-45da4ed056a8-kube-api-access-tzlfb\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:04.985320 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:04.985290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq58\" (UniqueName: \"kubernetes.io/projected/84c151bf-55cf-47a5-9b0c-c550b60f8633-kube-api-access-khq58\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.085784 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.085764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62c147fe-9e8b-4025-b734-45da4ed056a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:05.085873 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.085789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c151bf-55cf-47a5-9b0c-c550b60f8633-cert\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.085873 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.085815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlfb\" (UniqueName: \"kubernetes.io/projected/62c147fe-9e8b-4025-b734-45da4ed056a8-kube-api-access-tzlfb\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:05.085873 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.085830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq58\" (UniqueName: \"kubernetes.io/projected/84c151bf-55cf-47a5-9b0c-c550b60f8633-kube-api-access-khq58\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.086004 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:49:05.085908    2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 09 14:49:05.086004 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:49:05.085922    2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 09 14:49:05.086004 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:49:05.085963    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c147fe-9e8b-4025-b734-45da4ed056a8-cert podName:62c147fe-9e8b-4025-b734-45da4ed056a8 nodeName:}" failed. No retries permitted until 2026-07-09 14:49:05.585945642 +0000 UTC m=+546.619474120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/62c147fe-9e8b-4025-b734-45da4ed056a8-cert") pod "llmisvc-controller-manager-777c6bd7d5-xqz6t" (UID: "62c147fe-9e8b-4025-b734-45da4ed056a8") : secret "llmisvc-webhook-server-cert" not found
Jul 09 14:49:05.086004 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:49:05.085999    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84c151bf-55cf-47a5-9b0c-c550b60f8633-cert podName:84c151bf-55cf-47a5-9b0c-c550b60f8633 nodeName:}" failed. No retries permitted until 2026-07-09 14:49:05.585971037 +0000 UTC m=+546.619499497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/84c151bf-55cf-47a5-9b0c-c550b60f8633-cert") pod "kserve-controller-manager-d6f4b65b8-dq9wn" (UID: "84c151bf-55cf-47a5-9b0c-c550b60f8633") : secret "kserve-webhook-server-cert" not found
Jul 09 14:49:05.095336 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.095313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq58\" (UniqueName: \"kubernetes.io/projected/84c151bf-55cf-47a5-9b0c-c550b60f8633-kube-api-access-khq58\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.095412 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.095400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlfb\" (UniqueName: \"kubernetes.io/projected/62c147fe-9e8b-4025-b734-45da4ed056a8-kube-api-access-tzlfb\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:05.590688 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.590648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62c147fe-9e8b-4025-b734-45da4ed056a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:05.590688 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.590688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c151bf-55cf-47a5-9b0c-c550b60f8633-cert\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.593009 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.592970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c151bf-55cf-47a5-9b0c-c550b60f8633-cert\") pod \"kserve-controller-manager-d6f4b65b8-dq9wn\" (UID: \"84c151bf-55cf-47a5-9b0c-c550b60f8633\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.593119 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.593074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62c147fe-9e8b-4025-b734-45da4ed056a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xqz6t\" (UID: \"62c147fe-9e8b-4025-b734-45da4ed056a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:05.849403 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.849335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:05.855994 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.855957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:05.981832 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:05.981805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"]
Jul 09 14:49:05.984898 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:49:05.984868    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c151bf_55cf_47a5_9b0c_c550b60f8633.slice/crio-926a3b0ee8d8bd66322f3e61234f518a1c2af4d8191480389c557a408668ffeb WatchSource:0}: Error finding container 926a3b0ee8d8bd66322f3e61234f518a1c2af4d8191480389c557a408668ffeb: Status 404 returned error can't find the container with id 926a3b0ee8d8bd66322f3e61234f518a1c2af4d8191480389c557a408668ffeb
Jul 09 14:49:06.004914 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:06.004894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"]
Jul 09 14:49:06.006832 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:49:06.006813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod62c147fe_9e8b_4025_b734_45da4ed056a8.slice/crio-2a732a1cbb76eaa754d52718d6b5f660ae6e8d0c4579ead39c1956f396fb3002 WatchSource:0}: Error finding container 2a732a1cbb76eaa754d52718d6b5f660ae6e8d0c4579ead39c1956f396fb3002: Status 404 returned error can't find the container with id 2a732a1cbb76eaa754d52718d6b5f660ae6e8d0c4579ead39c1956f396fb3002
Jul 09 14:49:06.292062 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:06.292026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn" event={"ID":"84c151bf-55cf-47a5-9b0c-c550b60f8633","Type":"ContainerStarted","Data":"926a3b0ee8d8bd66322f3e61234f518a1c2af4d8191480389c557a408668ffeb"}
Jul 09 14:49:06.293442 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:06.293416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t" event={"ID":"62c147fe-9e8b-4025-b734-45da4ed056a8","Type":"ContainerStarted","Data":"2a732a1cbb76eaa754d52718d6b5f660ae6e8d0c4579ead39c1956f396fb3002"}
Jul 09 14:49:07.469352 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.469318    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"]
Jul 09 14:49:07.469766 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.469539    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v" podUID="8e275fb8-ee11-4d55-9d35-86dd661ede7a" containerName="discovery" containerID="cri-o://060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a" gracePeriod=30
Jul 09 14:49:07.719083 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.719048    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:49:07.808207 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808147    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e275fb8-ee11-4d55-9d35-86dd661ede7a-local-certs\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808207 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808192    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-kubeconfig\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808409 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-token\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808409 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-cacerts\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808409 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-ca-configmap\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808409 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-dns-cert\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808409 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6cbz\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-kube-api-access-j6cbz\") pod \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\" (UID: \"8e275fb8-ee11-4d55-9d35-86dd661ede7a\") "
Jul 09 14:49:07.808757 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.808721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 14:49:07.810645 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.810621    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-cacerts" (OuterVolumeSpecName: "cacerts") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:49:07.810645 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.810639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:49:07.810793 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.810772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:49:07.810838 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.810818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e275fb8-ee11-4d55-9d35-86dd661ede7a-local-certs" (OuterVolumeSpecName: "local-certs") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 14:49:07.810914 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.810898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:49:07.810952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.810928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-kube-api-access-j6cbz" (OuterVolumeSpecName: "kube-api-access-j6cbz") pod "8e275fb8-ee11-4d55-9d35-86dd661ede7a" (UID: "8e275fb8-ee11-4d55-9d35-86dd661ede7a"). InnerVolumeSpecName "kube-api-access-j6cbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:49:07.909033 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909006    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-kubeconfig\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:07.909033 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909028    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:07.909033 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909038    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-cacerts\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:07.909203 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909046    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-ca-configmap\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:07.909203 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909055    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e275fb8-ee11-4d55-9d35-86dd661ede7a-istio-csr-dns-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:07.909203 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909064    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6cbz\" (UniqueName: \"kubernetes.io/projected/8e275fb8-ee11-4d55-9d35-86dd661ede7a-kube-api-access-j6cbz\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:07.909203 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:07.909072    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e275fb8-ee11-4d55-9d35-86dd661ede7a-local-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:08.306326 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.306289    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e275fb8-ee11-4d55-9d35-86dd661ede7a" containerID="060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a" exitCode=0
Jul 09 14:49:08.306526 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.306337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v" event={"ID":"8e275fb8-ee11-4d55-9d35-86dd661ede7a","Type":"ContainerDied","Data":"060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a"}
Jul 09 14:49:08.306526 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.306348    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"
Jul 09 14:49:08.306526 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.306384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5m88v" event={"ID":"8e275fb8-ee11-4d55-9d35-86dd661ede7a","Type":"ContainerDied","Data":"d5b6c93c451552fd477ea010187f3c30879c5ec2dcfcc9d0c355255cc04e8964"}
Jul 09 14:49:08.306526 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.306405    2578 scope.go:117] "RemoveContainer" containerID="060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a"
Jul 09 14:49:08.332293 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.332267    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"]
Jul 09 14:49:08.337867 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.337840    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5m88v"]
Jul 09 14:49:08.631751 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.631475    2578 scope.go:117] "RemoveContainer" containerID="060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a"
Jul 09 14:49:08.632085 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:49:08.631788    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a\": container with ID starting with 060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a not found: ID does not exist" containerID="060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a"
Jul 09 14:49:08.632085 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:08.631816    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a"} err="failed to get container status \"060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a\": rpc error: code = NotFound desc = could not find container \"060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a\": container with ID starting with 060e37028d5eedcd5391e33becb92e0c957b850c6dad33b22f4d985de8e8ca9a not found: ID does not exist"
Jul 09 14:49:09.314676 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:09.314643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn" event={"ID":"84c151bf-55cf-47a5-9b0c-c550b60f8633","Type":"ContainerStarted","Data":"a9c2c7801b5d7f18d0482ec5b9077f42a0065d37345a218591e33bba4c179cc9"}
Jul 09 14:49:09.314801 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:09.314715    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:09.335180 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:09.335139    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn" podStartSLOduration=2.098127841 podStartE2EDuration="5.335122837s" totalImagesPullingTime="3.236994996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:09.334775568 +0000 UTC m=+550.368304077" watchObservedRunningTime="2026-07-09 14:49:09.335122837 +0000 UTC m=+550.368651319"
Jul 09 14:49:09.489883 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:09.489812    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e275fb8-ee11-4d55-9d35-86dd661ede7a" path="/var/lib/kubelet/pods/8e275fb8-ee11-4d55-9d35-86dd661ede7a/volumes"
Jul 09 14:49:10.321374 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:10.321339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t" event={"ID":"62c147fe-9e8b-4025-b734-45da4ed056a8","Type":"ContainerStarted","Data":"905db42c44aada60f17a7ed54f30a03f59edee51913a3d265bb80ffe53cd561e"}
Jul 09 14:49:10.321835 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:10.321559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:10.339380 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:10.339337    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t" podStartSLOduration=3.12033708 podStartE2EDuration="6.339324297s" totalImagesPullingTime="3.218987217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:10.338607825 +0000 UTC m=+551.372136307" watchObservedRunningTime="2026-07-09 14:49:10.339324297 +0000 UTC m=+551.372852779"
Jul 09 14:49:11.273318 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:11.273291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mks58"
Jul 09 14:49:11.275957 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:11.275888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8mht6"
Jul 09 14:49:40.326495 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:40.326465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dq9wn"
Jul 09 14:49:41.329335 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:41.329307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xqz6t"
Jul 09 14:49:55.596695 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.596661    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l488p"]
Jul 09 14:49:55.597098 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.596928    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e275fb8-ee11-4d55-9d35-86dd661ede7a" containerName="discovery"
Jul 09 14:49:55.597098 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.596939    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e275fb8-ee11-4d55-9d35-86dd661ede7a" containerName="discovery"
Jul 09 14:49:55.597098 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.597022    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e275fb8-ee11-4d55-9d35-86dd661ede7a" containerName="discovery"
Jul 09 14:49:55.599748 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.599731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:55.604105 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.604081    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 14:49:55.604676 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.604657    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 14:49:55.605620 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.605603    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bt8pj\""
Jul 09 14:49:55.625155 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.625130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l488p"]
Jul 09 14:49:55.747604 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.747564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44h6h\" (UniqueName: \"kubernetes.io/projected/70bb8e4c-f74e-4690-87cd-9ac7f047391a-kube-api-access-44h6h\") pod \"kuadrant-operator-catalog-l488p\" (UID: \"70bb8e4c-f74e-4690-87cd-9ac7f047391a\") " pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:55.848507 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.848434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44h6h\" (UniqueName: \"kubernetes.io/projected/70bb8e4c-f74e-4690-87cd-9ac7f047391a-kube-api-access-44h6h\") pod \"kuadrant-operator-catalog-l488p\" (UID: \"70bb8e4c-f74e-4690-87cd-9ac7f047391a\") " pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:55.876108 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.876084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44h6h\" (UniqueName: \"kubernetes.io/projected/70bb8e4c-f74e-4690-87cd-9ac7f047391a-kube-api-access-44h6h\") pod \"kuadrant-operator-catalog-l488p\" (UID: \"70bb8e4c-f74e-4690-87cd-9ac7f047391a\") " pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:55.908727 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.908693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:55.933044 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:55.933022    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l488p"]
Jul 09 14:49:56.039029 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.039006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l488p"]
Jul 09 14:49:56.041080 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:49:56.041053    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bb8e4c_f74e_4690_87cd_9ac7f047391a.slice/crio-327a4cd4661eb9d6c06431fc743d2202e0f728215a1f1707d5db605c09965569 WatchSource:0}: Error finding container 327a4cd4661eb9d6c06431fc743d2202e0f728215a1f1707d5db605c09965569: Status 404 returned error can't find the container with id 327a4cd4661eb9d6c06431fc743d2202e0f728215a1f1707d5db605c09965569
Jul 09 14:49:56.133545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.133482    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4lld"]
Jul 09 14:49:56.137664 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.137647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:49:56.149766 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.149745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4lld"]
Jul 09 14:49:56.251699 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.251671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwml\" (UniqueName: \"kubernetes.io/projected/b3513bed-14aa-4b7b-bb10-c193a9437c8b-kube-api-access-prwml\") pod \"kuadrant-operator-catalog-b4lld\" (UID: \"b3513bed-14aa-4b7b-bb10-c193a9437c8b\") " pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:49:56.352732 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.352702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwml\" (UniqueName: \"kubernetes.io/projected/b3513bed-14aa-4b7b-bb10-c193a9437c8b-kube-api-access-prwml\") pod \"kuadrant-operator-catalog-b4lld\" (UID: \"b3513bed-14aa-4b7b-bb10-c193a9437c8b\") " pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:49:56.363109 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.363079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwml\" (UniqueName: \"kubernetes.io/projected/b3513bed-14aa-4b7b-bb10-c193a9437c8b-kube-api-access-prwml\") pod \"kuadrant-operator-catalog-b4lld\" (UID: \"b3513bed-14aa-4b7b-bb10-c193a9437c8b\") " pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:49:56.445970 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.445940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:49:56.502335 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.502235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l488p" event={"ID":"70bb8e4c-f74e-4690-87cd-9ac7f047391a","Type":"ContainerStarted","Data":"327a4cd4661eb9d6c06431fc743d2202e0f728215a1f1707d5db605c09965569"}
Jul 09 14:49:56.572193 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:56.572170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4lld"]
Jul 09 14:49:56.573938 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:49:56.573914    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3513bed_14aa_4b7b_bb10_c193a9437c8b.slice/crio-73ed27410be97e7b4862e589a0ac2017b14d7687cfd319f1e28f6a185bfdc4cb WatchSource:0}: Error finding container 73ed27410be97e7b4862e589a0ac2017b14d7687cfd319f1e28f6a185bfdc4cb: Status 404 returned error can't find the container with id 73ed27410be97e7b4862e589a0ac2017b14d7687cfd319f1e28f6a185bfdc4cb
Jul 09 14:49:57.512126 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:57.512087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b4lld" event={"ID":"b3513bed-14aa-4b7b-bb10-c193a9437c8b","Type":"ContainerStarted","Data":"73ed27410be97e7b4862e589a0ac2017b14d7687cfd319f1e28f6a185bfdc4cb"}
Jul 09 14:49:58.517468 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.517370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b4lld" event={"ID":"b3513bed-14aa-4b7b-bb10-c193a9437c8b","Type":"ContainerStarted","Data":"fa05e07601fb7cdc36fd55424239d5eac90ef05d8f701f058a1e5f71fd00476b"}
Jul 09 14:49:58.518785 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.518761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l488p" event={"ID":"70bb8e4c-f74e-4690-87cd-9ac7f047391a","Type":"ContainerStarted","Data":"fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6"}
Jul 09 14:49:58.518905 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.518809    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-l488p" podUID="70bb8e4c-f74e-4690-87cd-9ac7f047391a" containerName="registry-server" containerID="cri-o://fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6" gracePeriod=2
Jul 09 14:49:58.585233 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.585194    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-l488p" podStartSLOduration=1.3864073829999999 podStartE2EDuration="3.585179293s" totalImagesPullingTime="2.19877191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:58.582535078 +0000 UTC m=+599.616063580" watchObservedRunningTime="2026-07-09 14:49:58.585179293 +0000 UTC m=+599.618707775"
Jul 09 14:49:58.585656 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.585628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-b4lld" podStartSLOduration=0.912389011 podStartE2EDuration="2.585623365s" totalImagesPullingTime="1.673234354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:58.547395644 +0000 UTC m=+599.580924129" watchObservedRunningTime="2026-07-09 14:49:58.585623365 +0000 UTC m=+599.619151846"
Jul 09 14:49:58.748011 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.747967    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:58.774868 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.774803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44h6h\" (UniqueName: \"kubernetes.io/projected/70bb8e4c-f74e-4690-87cd-9ac7f047391a-kube-api-access-44h6h\") pod \"70bb8e4c-f74e-4690-87cd-9ac7f047391a\" (UID: \"70bb8e4c-f74e-4690-87cd-9ac7f047391a\") "
Jul 09 14:49:58.776807 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.776783    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb8e4c-f74e-4690-87cd-9ac7f047391a-kube-api-access-44h6h" (OuterVolumeSpecName: "kube-api-access-44h6h") pod "70bb8e4c-f74e-4690-87cd-9ac7f047391a" (UID: "70bb8e4c-f74e-4690-87cd-9ac7f047391a"). InnerVolumeSpecName "kube-api-access-44h6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:49:58.875235 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:58.875213    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44h6h\" (UniqueName: \"kubernetes.io/projected/70bb8e4c-f74e-4690-87cd-9ac7f047391a-kube-api-access-44h6h\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:49:59.327443 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.327417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:49:59.327613 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.327480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:49:59.523947 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.523889    2578 generic.go:356] "Generic (PLEG): container finished" podID="70bb8e4c-f74e-4690-87cd-9ac7f047391a" containerID="fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6" exitCode=0
Jul 09 14:49:59.523947 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.523929    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-l488p"
Jul 09 14:49:59.524522 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.523957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l488p" event={"ID":"70bb8e4c-f74e-4690-87cd-9ac7f047391a","Type":"ContainerDied","Data":"fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6"}
Jul 09 14:49:59.524522 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.524015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-l488p" event={"ID":"70bb8e4c-f74e-4690-87cd-9ac7f047391a","Type":"ContainerDied","Data":"327a4cd4661eb9d6c06431fc743d2202e0f728215a1f1707d5db605c09965569"}
Jul 09 14:49:59.524522 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.524038    2578 scope.go:117] "RemoveContainer" containerID="fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6"
Jul 09 14:49:59.532189 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.532173    2578 scope.go:117] "RemoveContainer" containerID="fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6"
Jul 09 14:49:59.532438 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:49:59.532417    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6\": container with ID starting with fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6 not found: ID does not exist" containerID="fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6"
Jul 09 14:49:59.532503 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.532445    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6"} err="failed to get container status \"fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6\": rpc error: code = NotFound desc = could not find container \"fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6\": container with ID starting with fccdc6b97cef5b27b45a498d0079dba1870345fac3042eaf1cdbb1a768d57df6 not found: ID does not exist"
Jul 09 14:49:59.550587 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.550564    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l488p"]
Jul 09 14:49:59.560440 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:49:59.560419    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-l488p"]
Jul 09 14:50:01.488142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:01.488110    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bb8e4c-f74e-4690-87cd-9ac7f047391a" path="/var/lib/kubelet/pods/70bb8e4c-f74e-4690-87cd-9ac7f047391a/volumes"
Jul 09 14:50:06.446751 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:06.446711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:50:06.447253 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:06.446777    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:50:06.468897 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:06.468874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:50:06.573193 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:06.573151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-b4lld"
Jul 09 14:50:25.684050 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.683960    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"]
Jul 09 14:50:25.684410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.684250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bb8e4c-f74e-4690-87cd-9ac7f047391a" containerName="registry-server"
Jul 09 14:50:25.684410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.684260    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bb8e4c-f74e-4690-87cd-9ac7f047391a" containerName="registry-server"
Jul 09 14:50:25.684410 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.684311    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bb8e4c-f74e-4690-87cd-9ac7f047391a" containerName="registry-server"
Jul 09 14:50:25.687114 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.687098    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.690789 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.690766    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gf4fj\""
Jul 09 14:50:25.727957 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.727929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"]
Jul 09 14:50:25.763071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.763036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmglv\" (UniqueName: \"kubernetes.io/projected/b82b649f-8215-4540-b902-954b15ac23a0-kube-api-access-kmglv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.763071 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.763075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b82b649f-8215-4540-b902-954b15ac23a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.864245 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.864196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmglv\" (UniqueName: \"kubernetes.io/projected/b82b649f-8215-4540-b902-954b15ac23a0-kube-api-access-kmglv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.864245 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.864245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b82b649f-8215-4540-b902-954b15ac23a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.864591 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.864573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b82b649f-8215-4540-b902-954b15ac23a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.881143 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.881114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmglv\" (UniqueName: \"kubernetes.io/projected/b82b649f-8215-4540-b902-954b15ac23a0-kube-api-access-kmglv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:25.996950 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:25.996857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:26.152751 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:26.152716    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"]
Jul 09 14:50:26.156022 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:26.155994    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82b649f_8215_4540_b902_954b15ac23a0.slice/crio-0b8277c508c8efb06b7a155423b6c2af472ba59ee6e6a5de8871b61802deaa07 WatchSource:0}: Error finding container 0b8277c508c8efb06b7a155423b6c2af472ba59ee6e6a5de8871b61802deaa07: Status 404 returned error can't find the container with id 0b8277c508c8efb06b7a155423b6c2af472ba59ee6e6a5de8871b61802deaa07
Jul 09 14:50:26.158511 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:26.158494    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 14:50:26.635114 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:26.635080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l" event={"ID":"b82b649f-8215-4540-b902-954b15ac23a0","Type":"ContainerStarted","Data":"0b8277c508c8efb06b7a155423b6c2af472ba59ee6e6a5de8871b61802deaa07"}
Jul 09 14:50:28.289500 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.289465    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"]
Jul 09 14:50:28.294628 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.294610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:28.297942 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.297921    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wwttv\""
Jul 09 14:50:28.310610 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.310584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"]
Jul 09 14:50:28.385907 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.385870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdt6n\" (UniqueName: \"kubernetes.io/projected/1a2a78fd-09d9-4f62-820b-778acebd3656-kube-api-access-gdt6n\") pod \"authorino-operator-75cc8d5cb7-2v7h2\" (UID: \"1a2a78fd-09d9-4f62-820b-778acebd3656\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:28.491405 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.487210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdt6n\" (UniqueName: \"kubernetes.io/projected/1a2a78fd-09d9-4f62-820b-778acebd3656-kube-api-access-gdt6n\") pod \"authorino-operator-75cc8d5cb7-2v7h2\" (UID: \"1a2a78fd-09d9-4f62-820b-778acebd3656\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:28.498906 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.498875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdt6n\" (UniqueName: \"kubernetes.io/projected/1a2a78fd-09d9-4f62-820b-778acebd3656-kube-api-access-gdt6n\") pod \"authorino-operator-75cc8d5cb7-2v7h2\" (UID: \"1a2a78fd-09d9-4f62-820b-778acebd3656\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:28.604808 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.604731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:28.746690 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:28.746663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"]
Jul 09 14:50:28.747705 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:28.747667    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2a78fd_09d9_4f62_820b_778acebd3656.slice/crio-70f88d3969348812f482b5cac5369f1f4052ea2c0d42339d8cde9ef5dbe540b6 WatchSource:0}: Error finding container 70f88d3969348812f482b5cac5369f1f4052ea2c0d42339d8cde9ef5dbe540b6: Status 404 returned error can't find the container with id 70f88d3969348812f482b5cac5369f1f4052ea2c0d42339d8cde9ef5dbe540b6
Jul 09 14:50:29.651719 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:29.651654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2" event={"ID":"1a2a78fd-09d9-4f62-820b-778acebd3656","Type":"ContainerStarted","Data":"70f88d3969348812f482b5cac5369f1f4052ea2c0d42339d8cde9ef5dbe540b6"}
Jul 09 14:50:30.593214 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.593189    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"]
Jul 09 14:50:30.601656 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.601633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:30.605159 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.605138    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2ww2n\""
Jul 09 14:50:30.605486 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.605470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 09 14:50:30.647502 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.647467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"]
Jul 09 14:50:30.707341 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.707296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hx7\" (UniqueName: \"kubernetes.io/projected/da51c1cb-eda5-4134-919c-0fec24b68ade-kube-api-access-j9hx7\") pod \"dns-operator-controller-manager-864dd78c4c-d4hh5\" (UID: \"da51c1cb-eda5-4134-919c-0fec24b68ade\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:30.808262 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.808216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hx7\" (UniqueName: \"kubernetes.io/projected/da51c1cb-eda5-4134-919c-0fec24b68ade-kube-api-access-j9hx7\") pod \"dns-operator-controller-manager-864dd78c4c-d4hh5\" (UID: \"da51c1cb-eda5-4134-919c-0fec24b68ade\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:30.828252 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.828220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hx7\" (UniqueName: \"kubernetes.io/projected/da51c1cb-eda5-4134-919c-0fec24b68ade-kube-api-access-j9hx7\") pod \"dns-operator-controller-manager-864dd78c4c-d4hh5\" (UID: \"da51c1cb-eda5-4134-919c-0fec24b68ade\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:30.912686 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:30.912659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:32.239545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.239515    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"]
Jul 09 14:50:32.242500 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:32.242471    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda51c1cb_eda5_4134_919c_0fec24b68ade.slice/crio-6ee4c7e87d2f420a7a553d324423a37f5d7510375f1e7d4f54b00eaa1f4faa2f WatchSource:0}: Error finding container 6ee4c7e87d2f420a7a553d324423a37f5d7510375f1e7d4f54b00eaa1f4faa2f: Status 404 returned error can't find the container with id 6ee4c7e87d2f420a7a553d324423a37f5d7510375f1e7d4f54b00eaa1f4faa2f
Jul 09 14:50:32.668777 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.668743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l" event={"ID":"b82b649f-8215-4540-b902-954b15ac23a0","Type":"ContainerStarted","Data":"128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19"}
Jul 09 14:50:32.669092 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.668856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:32.670193 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.670169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5" event={"ID":"da51c1cb-eda5-4134-919c-0fec24b68ade","Type":"ContainerStarted","Data":"6ee4c7e87d2f420a7a553d324423a37f5d7510375f1e7d4f54b00eaa1f4faa2f"}
Jul 09 14:50:32.671598 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.671577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2" event={"ID":"1a2a78fd-09d9-4f62-820b-778acebd3656","Type":"ContainerStarted","Data":"631fc7bd6907c3538d8704e8b921983922ea02de85ba49e7f085f58e202b83c5"}
Jul 09 14:50:32.671691 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.671633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:32.725657 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.725599    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l" podStartSLOduration=1.7838511559999999 podStartE2EDuration="7.725582943s" totalImagesPullingTime="5.941731787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:32.717258435 +0000 UTC m=+633.750786935" watchObservedRunningTime="2026-07-09 14:50:32.725582943 +0000 UTC m=+633.759111423"
Jul 09 14:50:32.784563 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:32.784510    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2" podStartSLOduration=1.447978382 podStartE2EDuration="4.784491942s" totalImagesPullingTime="3.33651356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:32.783457382 +0000 UTC m=+633.816985878" watchObservedRunningTime="2026-07-09 14:50:32.784491942 +0000 UTC m=+633.818020424"
Jul 09 14:50:33.465046 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.465005    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"]
Jul 09 14:50:33.468373 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.468354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:33.472633 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.472611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tp684\""
Jul 09 14:50:33.489690 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.489659    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"]
Jul 09 14:50:33.533111 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.533070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzk24\" (UniqueName: \"kubernetes.io/projected/7003a33e-7cc0-475e-a8f3-3b18abeca687-kube-api-access-hzk24\") pod \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" (UID: \"7003a33e-7cc0-475e-a8f3-3b18abeca687\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:33.634494 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.634453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzk24\" (UniqueName: \"kubernetes.io/projected/7003a33e-7cc0-475e-a8f3-3b18abeca687-kube-api-access-hzk24\") pod \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" (UID: \"7003a33e-7cc0-475e-a8f3-3b18abeca687\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:33.647753 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.647718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzk24\" (UniqueName: \"kubernetes.io/projected/7003a33e-7cc0-475e-a8f3-3b18abeca687-kube-api-access-hzk24\") pod \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" (UID: \"7003a33e-7cc0-475e-a8f3-3b18abeca687\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:33.778968 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.778862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:33.936199 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:33.936175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"]
Jul 09 14:50:33.938052 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:33.937955    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7003a33e_7cc0_475e_a8f3_3b18abeca687.slice/crio-9e1a00001a08f4ee1aeae93c97a7eea94e6b6c623c6876770c34f993a7ce6801 WatchSource:0}: Error finding container 9e1a00001a08f4ee1aeae93c97a7eea94e6b6c623c6876770c34f993a7ce6801: Status 404 returned error can't find the container with id 9e1a00001a08f4ee1aeae93c97a7eea94e6b6c623c6876770c34f993a7ce6801
Jul 09 14:50:34.683291 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:34.683256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c" event={"ID":"7003a33e-7cc0-475e-a8f3-3b18abeca687","Type":"ContainerStarted","Data":"9e1a00001a08f4ee1aeae93c97a7eea94e6b6c623c6876770c34f993a7ce6801"}
Jul 09 14:50:35.689407 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:35.689380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5" event={"ID":"da51c1cb-eda5-4134-919c-0fec24b68ade","Type":"ContainerStarted","Data":"ab6b87e679d3a95629894bac890c8e92db64da76ff5dcb7abb17055e4fe9de99"}
Jul 09 14:50:35.689799 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:35.689498    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:35.719919 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:35.719874    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5" podStartSLOduration=3.155633749 podStartE2EDuration="5.719855954s" totalImagesPullingTime="2.564222205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:35.719595505 +0000 UTC m=+636.753124008" watchObservedRunningTime="2026-07-09 14:50:35.719855954 +0000 UTC m=+636.753384438"
Jul 09 14:50:36.694630 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:36.694591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c" event={"ID":"7003a33e-7cc0-475e-a8f3-3b18abeca687","Type":"ContainerStarted","Data":"d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58"}
Jul 09 14:50:36.695042 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:36.694701    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:36.721528 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:36.721477    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c" podStartSLOduration=1.998547137 podStartE2EDuration="3.721460181s" totalImagesPullingTime="1.722913044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:36.721378786 +0000 UTC m=+637.754907270" watchObservedRunningTime="2026-07-09 14:50:36.721460181 +0000 UTC m=+637.754988663"
Jul 09 14:50:43.678896 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:43.678865    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2v7h2"
Jul 09 14:50:43.679307 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:43.679040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.310756 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.310720    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"]
Jul 09 14:50:46.311142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.310894    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l" podUID="b82b649f-8215-4540-b902-954b15ac23a0" containerName="manager" containerID="cri-o://128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19" gracePeriod=2
Jul 09 14:50:46.324530 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.324504    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"]
Jul 09 14:50:46.393955 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.393475    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"]
Jul 09 14:50:46.393955 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.393909    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82b649f-8215-4540-b902-954b15ac23a0" containerName="manager"
Jul 09 14:50:46.393955 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.393926    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82b649f-8215-4540-b902-954b15ac23a0" containerName="manager"
Jul 09 14:50:46.394249 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.394034    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82b649f-8215-4540-b902-954b15ac23a0" containerName="manager"
Jul 09 14:50:46.397161 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.397137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"]
Jul 09 14:50:46.397287 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.397246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.397350 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.397303    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" containerName="manager" containerID="cri-o://d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58" gracePeriod=2
Jul 09 14:50:46.399452 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.399416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.414430 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.414292    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.425570 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.425541    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"]
Jul 09 14:50:46.439350 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.439301    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.440961 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.440937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"]
Jul 09 14:50:46.444074 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.444043    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.526013 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.525951    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"]
Jul 09 14:50:46.526394 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.526368    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" containerName="manager"
Jul 09 14:50:46.526394 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.526387    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" containerName="manager"
Jul 09 14:50:46.526548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.526469    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" containerName="manager"
Jul 09 14:50:46.529341 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.529320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:50:46.537431 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.537394    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.538746 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.538715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt85\" (UniqueName: \"kubernetes.io/projected/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-kube-api-access-ptt85\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zznvf\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.538862 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.538786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zznvf\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.540273 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.540248    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.547459 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.547435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"]
Jul 09 14:50:46.639699 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.639672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt85\" (UniqueName: \"kubernetes.io/projected/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-kube-api-access-ptt85\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zznvf\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.639788 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.639747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zznvf\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.639844 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.639792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpxd\" (UniqueName: \"kubernetes.io/projected/a0575a99-9a05-467a-90cd-9f3b43b8f73f-kube-api-access-2hpxd\") pod \"limitador-operator-controller-manager-859d7b78bd-29bvx\" (UID: \"a0575a99-9a05-467a-90cd-9f3b43b8f73f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:50:46.640235 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.640214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zznvf\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.645781 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.645762    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.648414 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.648397    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.654035 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.654007    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.657643 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.657591    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.662274 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.662254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt85\" (UniqueName: \"kubernetes.io/projected/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-kube-api-access-ptt85\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zznvf\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.663199 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.663177    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.670317 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.670294    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.697127 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.697101    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d4hh5"
Jul 09 14:50:46.714802 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.714773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:46.737347 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.737315    2578 generic.go:356] "Generic (PLEG): container finished" podID="7003a33e-7cc0-475e-a8f3-3b18abeca687" containerID="d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58" exitCode=0
Jul 09 14:50:46.737485 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.737376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.737485 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.737402    2578 scope.go:117] "RemoveContainer" containerID="d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58"
Jul 09 14:50:46.738709 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.738681    2578 generic.go:356] "Generic (PLEG): container finished" podID="b82b649f-8215-4540-b902-954b15ac23a0" containerID="128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19" exitCode=0
Jul 09 14:50:46.738815 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.738733    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.740960 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.740934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b82b649f-8215-4540-b902-954b15ac23a0-extensions-socket-volume\") pod \"b82b649f-8215-4540-b902-954b15ac23a0\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") "
Jul 09 14:50:46.741082 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.741019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzk24\" (UniqueName: \"kubernetes.io/projected/7003a33e-7cc0-475e-a8f3-3b18abeca687-kube-api-access-hzk24\") pod \"7003a33e-7cc0-475e-a8f3-3b18abeca687\" (UID: \"7003a33e-7cc0-475e-a8f3-3b18abeca687\") "
Jul 09 14:50:46.741082 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.741073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmglv\" (UniqueName: \"kubernetes.io/projected/b82b649f-8215-4540-b902-954b15ac23a0-kube-api-access-kmglv\") pod \"b82b649f-8215-4540-b902-954b15ac23a0\" (UID: \"b82b649f-8215-4540-b902-954b15ac23a0\") "
Jul 09 14:50:46.741324 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.741303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpxd\" (UniqueName: \"kubernetes.io/projected/a0575a99-9a05-467a-90cd-9f3b43b8f73f-kube-api-access-2hpxd\") pod \"limitador-operator-controller-manager-859d7b78bd-29bvx\" (UID: \"a0575a99-9a05-467a-90cd-9f3b43b8f73f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:50:46.741472 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.741450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b649f-8215-4540-b902-954b15ac23a0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b82b649f-8215-4540-b902-954b15ac23a0" (UID: "b82b649f-8215-4540-b902-954b15ac23a0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 14:50:46.743599 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.743573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82b649f-8215-4540-b902-954b15ac23a0-kube-api-access-kmglv" (OuterVolumeSpecName: "kube-api-access-kmglv") pod "b82b649f-8215-4540-b902-954b15ac23a0" (UID: "b82b649f-8215-4540-b902-954b15ac23a0"). InnerVolumeSpecName "kube-api-access-kmglv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:50:46.743694 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.743595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7003a33e-7cc0-475e-a8f3-3b18abeca687-kube-api-access-hzk24" (OuterVolumeSpecName: "kube-api-access-hzk24") pod "7003a33e-7cc0-475e-a8f3-3b18abeca687" (UID: "7003a33e-7cc0-475e-a8f3-3b18abeca687"). InnerVolumeSpecName "kube-api-access-hzk24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:50:46.745832 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.745813    2578 scope.go:117] "RemoveContainer" containerID="d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58"
Jul 09 14:50:46.746204 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.746185    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58\": container with ID starting with d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58 not found: ID does not exist" containerID="d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58"
Jul 09 14:50:46.746262 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.746211    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58"} err="failed to get container status \"d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58\": rpc error: code = NotFound desc = could not find container \"d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58\": container with ID starting with d0e60c6cee1f9e87f2c5ba9770eed64707b45a60c2f88569ece9c62b9b42fe58 not found: ID does not exist"
Jul 09 14:50:46.746262 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.746227    2578 scope.go:117] "RemoveContainer" containerID="128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19"
Jul 09 14:50:46.754563 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.754545    2578 scope.go:117] "RemoveContainer" containerID="128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19"
Jul 09 14:50:46.754886 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.754864    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19\": container with ID starting with 128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19 not found: ID does not exist" containerID="128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19"
Jul 09 14:50:46.754998 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.754891    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19"} err="failed to get container status \"128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19\": rpc error: code = NotFound desc = could not find container \"128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19\": container with ID starting with 128ecd9731dd471c7f8b228d8803bfc8ba68b42b126d2bfc0c9cc8df08c22f19 not found: ID does not exist"
Jul 09 14:50:46.769135 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.768950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpxd\" (UniqueName: \"kubernetes.io/projected/a0575a99-9a05-467a-90cd-9f3b43b8f73f-kube-api-access-2hpxd\") pod \"limitador-operator-controller-manager-859d7b78bd-29bvx\" (UID: \"a0575a99-9a05-467a-90cd-9f3b43b8f73f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:50:46.774136 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.774101    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.777583 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.777486    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.781926 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.781894    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:46.784954 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:46.784933    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:46.841126 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.840281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:50:46.842681 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.842651    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b82b649f-8215-4540-b902-954b15ac23a0-extensions-socket-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:50:46.842681 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.842674    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzk24\" (UniqueName: \"kubernetes.io/projected/7003a33e-7cc0-475e-a8f3-3b18abeca687-kube-api-access-hzk24\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:50:46.842848 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.842684    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmglv\" (UniqueName: \"kubernetes.io/projected/b82b649f-8215-4540-b902-954b15ac23a0-kube-api-access-kmglv\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:50:46.877951 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.877922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"]
Jul 09 14:50:46.882292 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:46.882261    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac06b34_df65_41d2_a0bf_793bfeeafbdc.slice/crio-3b4a424d6567cd1ebb307830e88f4019a359e53357e64f84ad7cd28f46613c18 WatchSource:0}: Error finding container 3b4a424d6567cd1ebb307830e88f4019a359e53357e64f84ad7cd28f46613c18: Status 404 returned error can't find the container with id 3b4a424d6567cd1ebb307830e88f4019a359e53357e64f84ad7cd28f46613c18
Jul 09 14:50:46.995834 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:46.995804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"]
Jul 09 14:50:46.998776 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:46.998749    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0575a99_9a05_467a_90cd_9f3b43b8f73f.slice/crio-6b09e9705496abd57e0b3f0c11cecbc31a16cd47d2a1793d819363ed22f56ba0 WatchSource:0}: Error finding container 6b09e9705496abd57e0b3f0c11cecbc31a16cd47d2a1793d819363ed22f56ba0: Status 404 returned error can't find the container with id 6b09e9705496abd57e0b3f0c11cecbc31a16cd47d2a1793d819363ed22f56ba0
Jul 09 14:50:47.054003 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:47.053958    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:47.058205 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:47.058173    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:47.068614 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:47.068586    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gpl2c\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gpl2c"
Jul 09 14:50:47.092201 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:50:47.092132    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4px4l\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="b82b649f-8215-4540-b902-954b15ac23a0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4px4l"
Jul 09 14:50:47.488624 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.488590    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7003a33e-7cc0-475e-a8f3-3b18abeca687" path="/var/lib/kubelet/pods/7003a33e-7cc0-475e-a8f3-3b18abeca687/volumes"
Jul 09 14:50:47.488994 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.488922    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82b649f-8215-4540-b902-954b15ac23a0" path="/var/lib/kubelet/pods/b82b649f-8215-4540-b902-954b15ac23a0/volumes"
Jul 09 14:50:47.746631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.746521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx" event={"ID":"a0575a99-9a05-467a-90cd-9f3b43b8f73f","Type":"ContainerStarted","Data":"891f13a65d4aab27980351027c401acf178207dc93038ec58d6ed2af7526ce31"}
Jul 09 14:50:47.746631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.746582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx" event={"ID":"a0575a99-9a05-467a-90cd-9f3b43b8f73f","Type":"ContainerStarted","Data":"6b09e9705496abd57e0b3f0c11cecbc31a16cd47d2a1793d819363ed22f56ba0"}
Jul 09 14:50:47.746631 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.746613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:50:47.748132 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.748107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf" event={"ID":"1ac06b34-df65-41d2-a0bf-793bfeeafbdc","Type":"ContainerStarted","Data":"4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8"}
Jul 09 14:50:47.748230 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.748151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:47.748230 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.748166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf" event={"ID":"1ac06b34-df65-41d2-a0bf-793bfeeafbdc","Type":"ContainerStarted","Data":"3b4a424d6567cd1ebb307830e88f4019a359e53357e64f84ad7cd28f46613c18"}
Jul 09 14:50:47.798020 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.797954    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx" podStartSLOduration=1.797938483 podStartE2EDuration="1.797938483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:47.796656744 +0000 UTC m=+648.830185238" watchObservedRunningTime="2026-07-09 14:50:47.797938483 +0000 UTC m=+648.831466965"
Jul 09 14:50:47.858924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:47.858871    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf" podStartSLOduration=1.85885408 podStartE2EDuration="1.85885408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:47.857742751 +0000 UTC m=+648.891271235" watchObservedRunningTime="2026-07-09 14:50:47.85885408 +0000 UTC m=+648.892382562"
Jul 09 14:50:49.532508 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.532474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"]
Jul 09 14:50:49.535956 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.535938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.574892 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.574857    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"]
Jul 09 14:50:49.665029 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.664996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead4ca74-7218-4116-ab54-a6fb55bd49e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-75jx2\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.665199 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.665071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6f9x\" (UniqueName: \"kubernetes.io/projected/ead4ca74-7218-4116-ab54-a6fb55bd49e6-kube-api-access-n6f9x\") pod \"kuadrant-operator-controller-manager-86878c769f-75jx2\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.765446 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.765409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead4ca74-7218-4116-ab54-a6fb55bd49e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-75jx2\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.765617 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.765502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6f9x\" (UniqueName: \"kubernetes.io/projected/ead4ca74-7218-4116-ab54-a6fb55bd49e6-kube-api-access-n6f9x\") pod \"kuadrant-operator-controller-manager-86878c769f-75jx2\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.765761 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.765741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead4ca74-7218-4116-ab54-a6fb55bd49e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-75jx2\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.794750 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.794684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6f9x\" (UniqueName: \"kubernetes.io/projected/ead4ca74-7218-4116-ab54-a6fb55bd49e6-kube-api-access-n6f9x\") pod \"kuadrant-operator-controller-manager-86878c769f-75jx2\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.845622 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.845585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:49.996669 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:49.996638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"]
Jul 09 14:50:50.000473 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:50:50.000448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead4ca74_7218_4116_ab54_a6fb55bd49e6.slice/crio-ba9567ad6497685f313810619a92b9eb567cd5a6eff8eb603bbefe033ee1a771 WatchSource:0}: Error finding container ba9567ad6497685f313810619a92b9eb567cd5a6eff8eb603bbefe033ee1a771: Status 404 returned error can't find the container with id ba9567ad6497685f313810619a92b9eb567cd5a6eff8eb603bbefe033ee1a771
Jul 09 14:50:50.762870 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:50.762836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2" event={"ID":"ead4ca74-7218-4116-ab54-a6fb55bd49e6","Type":"ContainerStarted","Data":"08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc"}
Jul 09 14:50:50.763271 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:50.762887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2" event={"ID":"ead4ca74-7218-4116-ab54-a6fb55bd49e6","Type":"ContainerStarted","Data":"ba9567ad6497685f313810619a92b9eb567cd5a6eff8eb603bbefe033ee1a771"}
Jul 09 14:50:50.763271 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:50.762960    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:50:50.791859 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:50.791806    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2" podStartSLOduration=1.791789402 podStartE2EDuration="1.791789402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:50.789494211 +0000 UTC m=+651.823022710" watchObservedRunningTime="2026-07-09 14:50:50.791789402 +0000 UTC m=+651.825317884"
Jul 09 14:50:58.754680 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:58.754644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:50:58.755189 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:50:58.754734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-29bvx"
Jul 09 14:51:01.769471 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:01.769431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 14:51:01.832200 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:01.832160    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"]
Jul 09 14:51:01.832442 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:01.832416    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf" podUID="1ac06b34-df65-41d2-a0bf-793bfeeafbdc" containerName="manager" containerID="cri-o://4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8" gracePeriod=10
Jul 09 14:51:02.074311 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.074288    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:51:02.168731 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.168695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-extensions-socket-volume\") pod \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") "
Jul 09 14:51:02.168911 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.168776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptt85\" (UniqueName: \"kubernetes.io/projected/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-kube-api-access-ptt85\") pod \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\" (UID: \"1ac06b34-df65-41d2-a0bf-793bfeeafbdc\") "
Jul 09 14:51:02.169106 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.169079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1ac06b34-df65-41d2-a0bf-793bfeeafbdc" (UID: "1ac06b34-df65-41d2-a0bf-793bfeeafbdc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 14:51:02.170749 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.170727    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-kube-api-access-ptt85" (OuterVolumeSpecName: "kube-api-access-ptt85") pod "1ac06b34-df65-41d2-a0bf-793bfeeafbdc" (UID: "1ac06b34-df65-41d2-a0bf-793bfeeafbdc"). InnerVolumeSpecName "kube-api-access-ptt85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:51:02.269836 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.269788    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt85\" (UniqueName: \"kubernetes.io/projected/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-kube-api-access-ptt85\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:02.269836 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.269829    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ac06b34-df65-41d2-a0bf-793bfeeafbdc-extensions-socket-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:02.816012 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.815953    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ac06b34-df65-41d2-a0bf-793bfeeafbdc" containerID="4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8" exitCode=0
Jul 09 14:51:02.816460 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.816040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"
Jul 09 14:51:02.816460 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.816042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf" event={"ID":"1ac06b34-df65-41d2-a0bf-793bfeeafbdc","Type":"ContainerDied","Data":"4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8"}
Jul 09 14:51:02.816460 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.816110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf" event={"ID":"1ac06b34-df65-41d2-a0bf-793bfeeafbdc","Type":"ContainerDied","Data":"3b4a424d6567cd1ebb307830e88f4019a359e53357e64f84ad7cd28f46613c18"}
Jul 09 14:51:02.816460 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.816142    2578 scope.go:117] "RemoveContainer" containerID="4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8"
Jul 09 14:51:02.825015 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.824992    2578 scope.go:117] "RemoveContainer" containerID="4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8"
Jul 09 14:51:02.825312 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:51:02.825293    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8\": container with ID starting with 4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8 not found: ID does not exist" containerID="4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8"
Jul 09 14:51:02.825387 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.825323    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8"} err="failed to get container status \"4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8\": rpc error: code = NotFound desc = could not find container \"4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8\": container with ID starting with 4633a8851625c9c5eb675c671f204d2fbdfcf726bc3a2fd828361ed9e0467da8 not found: ID does not exist"
Jul 09 14:51:02.862701 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.862662    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"]
Jul 09 14:51:02.867763 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:02.867735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zznvf"]
Jul 09 14:51:03.488307 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:03.488272    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac06b34-df65-41d2-a0bf-793bfeeafbdc" path="/var/lib/kubelet/pods/1ac06b34-df65-41d2-a0bf-793bfeeafbdc/volumes"
Jul 09 14:51:20.091824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.091786    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:20.092307 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.092139    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac06b34-df65-41d2-a0bf-793bfeeafbdc" containerName="manager"
Jul 09 14:51:20.092307 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.092151    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac06b34-df65-41d2-a0bf-793bfeeafbdc" containerName="manager"
Jul 09 14:51:20.092307 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.092205    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac06b34-df65-41d2-a0bf-793bfeeafbdc" containerName="manager"
Jul 09 14:51:20.100091 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.100069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.103333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.103296    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nds7q\""
Jul 09 14:51:20.103832 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.103682    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 14:51:20.110697 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.110676    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:20.173142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.173111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:20.200121 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.200091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4n2m\" (UniqueName: \"kubernetes.io/projected/60c91210-75c2-457e-bfe6-49ff17634f18-kube-api-access-r4n2m\") pod \"limitador-limitador-64c94fd64f-sswkr\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.200317 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.200156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60c91210-75c2-457e-bfe6-49ff17634f18-config-file\") pod \"limitador-limitador-64c94fd64f-sswkr\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.300545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.300509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4n2m\" (UniqueName: \"kubernetes.io/projected/60c91210-75c2-457e-bfe6-49ff17634f18-kube-api-access-r4n2m\") pod \"limitador-limitador-64c94fd64f-sswkr\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.300720 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.300570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60c91210-75c2-457e-bfe6-49ff17634f18-config-file\") pod \"limitador-limitador-64c94fd64f-sswkr\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.301240 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.301221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60c91210-75c2-457e-bfe6-49ff17634f18-config-file\") pod \"limitador-limitador-64c94fd64f-sswkr\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.314378 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.314350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4n2m\" (UniqueName: \"kubernetes.io/projected/60c91210-75c2-457e-bfe6-49ff17634f18-kube-api-access-r4n2m\") pod \"limitador-limitador-64c94fd64f-sswkr\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.411706 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.411676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:20.571910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.571879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:20.572148 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:51:20.572123    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c91210_75c2_457e_bfe6_49ff17634f18.slice/crio-2bde4b34e07aeea8b273a43bdde13162267caa2982450e7de63db49287f69208 WatchSource:0}: Error finding container 2bde4b34e07aeea8b273a43bdde13162267caa2982450e7de63db49287f69208: Status 404 returned error can't find the container with id 2bde4b34e07aeea8b273a43bdde13162267caa2982450e7de63db49287f69208
Jul 09 14:51:20.894189 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:20.894113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr" event={"ID":"60c91210-75c2-457e-bfe6-49ff17634f18","Type":"ContainerStarted","Data":"2bde4b34e07aeea8b273a43bdde13162267caa2982450e7de63db49287f69208"}
Jul 09 14:51:21.057605 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.057567    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-s8f2d"]
Jul 09 14:51:21.062278 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.062253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:21.065368 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.065345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p6qjh\""
Jul 09 14:51:21.070396 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.070360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8f2d"]
Jul 09 14:51:21.105566 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.105536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjn9\" (UniqueName: \"kubernetes.io/projected/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a-kube-api-access-lnjn9\") pod \"authorino-58cd955675-s8f2d\" (UID: \"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a\") " pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:21.206680 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.206643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjn9\" (UniqueName: \"kubernetes.io/projected/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a-kube-api-access-lnjn9\") pod \"authorino-58cd955675-s8f2d\" (UID: \"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a\") " pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:21.217555 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.217519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjn9\" (UniqueName: \"kubernetes.io/projected/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a-kube-api-access-lnjn9\") pod \"authorino-58cd955675-s8f2d\" (UID: \"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a\") " pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:21.372855 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.372821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:21.562388 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.562349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8f2d"]
Jul 09 14:51:21.564401 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:51:21.564355    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b41379b_a0ae_4c13_a7fb_c66af11d5b4a.slice/crio-7b7c6290643011bd98363091a236bb73dc6f4f8aa2b6f7bf5ac9e02947ce1478 WatchSource:0}: Error finding container 7b7c6290643011bd98363091a236bb73dc6f4f8aa2b6f7bf5ac9e02947ce1478: Status 404 returned error can't find the container with id 7b7c6290643011bd98363091a236bb73dc6f4f8aa2b6f7bf5ac9e02947ce1478
Jul 09 14:51:21.900055 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:21.899960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8f2d" event={"ID":"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a","Type":"ContainerStarted","Data":"7b7c6290643011bd98363091a236bb73dc6f4f8aa2b6f7bf5ac9e02947ce1478"}
Jul 09 14:51:23.911649 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:23.911610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr" event={"ID":"60c91210-75c2-457e-bfe6-49ff17634f18","Type":"ContainerStarted","Data":"b1ac2cd34a071d527fedca6662eb79b53f59aec5e46f70a72e488d0ea4e31e63"}
Jul 09 14:51:23.912115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:23.911802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:23.947121 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:23.947059    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr" podStartSLOduration=1.183799079 podStartE2EDuration="3.946950347s" totalImagesPullingTime="2.763151268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:51:23.945355186 +0000 UTC m=+684.978883684" watchObservedRunningTime="2026-07-09 14:51:23.946950347 +0000 UTC m=+684.980478832"
Jul 09 14:51:25.924207 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:25.924172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8f2d" event={"ID":"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a","Type":"ContainerStarted","Data":"b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903"}
Jul 09 14:51:25.943914 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:25.943863    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-s8f2d" podStartSLOduration=0.784526426 podStartE2EDuration="4.943847733s" totalImagesPullingTime="4.159321307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:51:25.942545255 +0000 UTC m=+686.976073737" watchObservedRunningTime="2026-07-09 14:51:25.943847733 +0000 UTC m=+686.977376287"
Jul 09 14:51:26.516831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:26.516801    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8f2d"]
Jul 09 14:51:27.932881 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:27.932820    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-s8f2d" podUID="5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" containerName="authorino" containerID="cri-o://b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903" gracePeriod=30
Jul 09 14:51:28.169008 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.168967    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:28.263519 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.263418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnjn9\" (UniqueName: \"kubernetes.io/projected/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a-kube-api-access-lnjn9\") pod \"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a\" (UID: \"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a\") "
Jul 09 14:51:28.265510 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.265484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a-kube-api-access-lnjn9" (OuterVolumeSpecName: "kube-api-access-lnjn9") pod "5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" (UID: "5b41379b-a0ae-4c13-a7fb-c66af11d5b4a"). InnerVolumeSpecName "kube-api-access-lnjn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:51:28.364170 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.364123    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnjn9\" (UniqueName: \"kubernetes.io/projected/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a-kube-api-access-lnjn9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:28.938682 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.938647    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" containerID="b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903" exitCode=0
Jul 09 14:51:28.939115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.938693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8f2d" event={"ID":"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a","Type":"ContainerDied","Data":"b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903"}
Jul 09 14:51:28.939115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.938698    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8f2d"
Jul 09 14:51:28.939115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.938727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8f2d" event={"ID":"5b41379b-a0ae-4c13-a7fb-c66af11d5b4a","Type":"ContainerDied","Data":"7b7c6290643011bd98363091a236bb73dc6f4f8aa2b6f7bf5ac9e02947ce1478"}
Jul 09 14:51:28.939115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.938744    2578 scope.go:117] "RemoveContainer" containerID="b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903"
Jul 09 14:51:28.946935 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.946917    2578 scope.go:117] "RemoveContainer" containerID="b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903"
Jul 09 14:51:28.947204 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:51:28.947184    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903\": container with ID starting with b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903 not found: ID does not exist" containerID="b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903"
Jul 09 14:51:28.947283 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.947212    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903"} err="failed to get container status \"b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903\": rpc error: code = NotFound desc = could not find container \"b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903\": container with ID starting with b087cffdd2fd7ce8e13685617123b60ebf4921b8ae1283c1c7c7ded83f043903 not found: ID does not exist"
Jul 09 14:51:28.962510 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.962485    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8f2d"]
Jul 09 14:51:28.968555 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:28.968532    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8f2d"]
Jul 09 14:51:29.489333 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:29.489296    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" path="/var/lib/kubelet/pods/5b41379b-a0ae-4c13-a7fb-c66af11d5b4a/volumes"
Jul 09 14:51:34.919734 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:34.919706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:35.515597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:35.515562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:35.515793 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:35.515742    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr" podUID="60c91210-75c2-457e-bfe6-49ff17634f18" containerName="limitador" containerID="cri-o://b1ac2cd34a071d527fedca6662eb79b53f59aec5e46f70a72e488d0ea4e31e63" gracePeriod=30
Jul 09 14:51:35.971169 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:35.971129    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c91210-75c2-457e-bfe6-49ff17634f18" containerID="b1ac2cd34a071d527fedca6662eb79b53f59aec5e46f70a72e488d0ea4e31e63" exitCode=0
Jul 09 14:51:35.971540 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:35.971179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr" event={"ID":"60c91210-75c2-457e-bfe6-49ff17634f18","Type":"ContainerDied","Data":"b1ac2cd34a071d527fedca6662eb79b53f59aec5e46f70a72e488d0ea4e31e63"}
Jul 09 14:51:36.058872 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.058842    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:36.125495 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.125404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60c91210-75c2-457e-bfe6-49ff17634f18-config-file\") pod \"60c91210-75c2-457e-bfe6-49ff17634f18\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") "
Jul 09 14:51:36.125495 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.125467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4n2m\" (UniqueName: \"kubernetes.io/projected/60c91210-75c2-457e-bfe6-49ff17634f18-kube-api-access-r4n2m\") pod \"60c91210-75c2-457e-bfe6-49ff17634f18\" (UID: \"60c91210-75c2-457e-bfe6-49ff17634f18\") "
Jul 09 14:51:36.125831 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.125805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c91210-75c2-457e-bfe6-49ff17634f18-config-file" (OuterVolumeSpecName: "config-file") pod "60c91210-75c2-457e-bfe6-49ff17634f18" (UID: "60c91210-75c2-457e-bfe6-49ff17634f18"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 14:51:36.127569 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.127543    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c91210-75c2-457e-bfe6-49ff17634f18-kube-api-access-r4n2m" (OuterVolumeSpecName: "kube-api-access-r4n2m") pod "60c91210-75c2-457e-bfe6-49ff17634f18" (UID: "60c91210-75c2-457e-bfe6-49ff17634f18"). InnerVolumeSpecName "kube-api-access-r4n2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:51:36.226367 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.226332    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60c91210-75c2-457e-bfe6-49ff17634f18-config-file\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:36.226367 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.226367    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4n2m\" (UniqueName: \"kubernetes.io/projected/60c91210-75c2-457e-bfe6-49ff17634f18-kube-api-access-r4n2m\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:36.977018 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.976965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr" event={"ID":"60c91210-75c2-457e-bfe6-49ff17634f18","Type":"ContainerDied","Data":"2bde4b34e07aeea8b273a43bdde13162267caa2982450e7de63db49287f69208"}
Jul 09 14:51:36.977481 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.977051    2578 scope.go:117] "RemoveContainer" containerID="b1ac2cd34a071d527fedca6662eb79b53f59aec5e46f70a72e488d0ea4e31e63"
Jul 09 14:51:36.977481 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.976968    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sswkr"
Jul 09 14:51:36.999719 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:36.999694    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:37.003011 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:37.002971    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sswkr"]
Jul 09 14:51:37.489032 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:37.488972    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c91210-75c2-457e-bfe6-49ff17634f18" path="/var/lib/kubelet/pods/60c91210-75c2-457e-bfe6-49ff17634f18/volumes"
Jul 09 14:51:41.786634 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.786603    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"]
Jul 09 14:51:41.787065 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.786940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" containerName="authorino"
Jul 09 14:51:41.787065 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.786956    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" containerName="authorino"
Jul 09 14:51:41.787065 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.786997    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c91210-75c2-457e-bfe6-49ff17634f18" containerName="limitador"
Jul 09 14:51:41.787065 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.787002    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c91210-75c2-457e-bfe6-49ff17634f18" containerName="limitador"
Jul 09 14:51:41.787065 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.787060    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c91210-75c2-457e-bfe6-49ff17634f18" containerName="limitador"
Jul 09 14:51:41.787065 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.787068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b41379b-a0ae-4c13-a7fb-c66af11d5b4a" containerName="authorino"
Jul 09 14:51:41.791330 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.791313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:41.794387 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.794358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 09 14:51:41.794491 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.794362    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 14:51:41.795721 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.795702    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 14:51:41.795850 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.795706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6dvtr\""
Jul 09 14:51:41.798999 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.798950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"]
Jul 09 14:51:41.872850 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.872806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e41d28c5-810a-41cd-832f-bcff0e86b2d6-data\") pod \"postgres-69fb4594cf-k62qv\" (UID: \"e41d28c5-810a-41cd-832f-bcff0e86b2d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:41.872850 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.872855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8cq\" (UniqueName: \"kubernetes.io/projected/e41d28c5-810a-41cd-832f-bcff0e86b2d6-kube-api-access-8j8cq\") pod \"postgres-69fb4594cf-k62qv\" (UID: \"e41d28c5-810a-41cd-832f-bcff0e86b2d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:41.973577 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.973544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e41d28c5-810a-41cd-832f-bcff0e86b2d6-data\") pod \"postgres-69fb4594cf-k62qv\" (UID: \"e41d28c5-810a-41cd-832f-bcff0e86b2d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:41.973577 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.973583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8cq\" (UniqueName: \"kubernetes.io/projected/e41d28c5-810a-41cd-832f-bcff0e86b2d6-kube-api-access-8j8cq\") pod \"postgres-69fb4594cf-k62qv\" (UID: \"e41d28c5-810a-41cd-832f-bcff0e86b2d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:41.973910 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.973891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e41d28c5-810a-41cd-832f-bcff0e86b2d6-data\") pod \"postgres-69fb4594cf-k62qv\" (UID: \"e41d28c5-810a-41cd-832f-bcff0e86b2d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:41.983524 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:41.983501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8cq\" (UniqueName: \"kubernetes.io/projected/e41d28c5-810a-41cd-832f-bcff0e86b2d6-kube-api-access-8j8cq\") pod \"postgres-69fb4594cf-k62qv\" (UID: \"e41d28c5-810a-41cd-832f-bcff0e86b2d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:42.102963 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:42.102848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:42.231400 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:42.231376    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"]
Jul 09 14:51:42.234145 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:51:42.234106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41d28c5_810a_41cd_832f_bcff0e86b2d6.slice/crio-88a6ecd4683faa3806ba6092746b1134c167a3de02789e6267779932aab285c6 WatchSource:0}: Error finding container 88a6ecd4683faa3806ba6092746b1134c167a3de02789e6267779932aab285c6: Status 404 returned error can't find the container with id 88a6ecd4683faa3806ba6092746b1134c167a3de02789e6267779932aab285c6
Jul 09 14:51:43.005368 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:43.005331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv" event={"ID":"e41d28c5-810a-41cd-832f-bcff0e86b2d6","Type":"ContainerStarted","Data":"88a6ecd4683faa3806ba6092746b1134c167a3de02789e6267779932aab285c6"}
Jul 09 14:51:48.039317 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:48.039237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv" event={"ID":"e41d28c5-810a-41cd-832f-bcff0e86b2d6","Type":"ContainerStarted","Data":"623f82c8c4892384995639d2ab2b5c71dc6418193f27d2509a4252df35629dae"}
Jul 09 14:51:48.039654 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:48.039371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:48.059260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:48.059209    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv" podStartSLOduration=1.5126493509999999 podStartE2EDuration="7.059195428s" totalImagesPullingTime="5.546546077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:51:48.057561612 +0000 UTC m=+709.091090112" watchObservedRunningTime="2026-07-09 14:51:48.059195428 +0000 UTC m=+709.092723910"
Jul 09 14:51:54.075697 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.075614    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k62qv"
Jul 09 14:51:54.620049 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.620013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sm52"]
Jul 09 14:51:54.623237 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.623212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:54.625970 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.625949    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p6qjh\""
Jul 09 14:51:54.634095 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.634074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sm52"]
Jul 09 14:51:54.679846 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.679817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2j7\" (UniqueName: \"kubernetes.io/projected/93befab7-20fb-465f-97f0-b5fe1309113c-kube-api-access-vz2j7\") pod \"authorino-5d4b67c446-4sm52\" (UID: \"93befab7-20fb-465f-97f0-b5fe1309113c\") " pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:54.780695 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.780649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2j7\" (UniqueName: \"kubernetes.io/projected/93befab7-20fb-465f-97f0-b5fe1309113c-kube-api-access-vz2j7\") pod \"authorino-5d4b67c446-4sm52\" (UID: \"93befab7-20fb-465f-97f0-b5fe1309113c\") " pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:54.790437 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.790390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2j7\" (UniqueName: \"kubernetes.io/projected/93befab7-20fb-465f-97f0-b5fe1309113c-kube-api-access-vz2j7\") pod \"authorino-5d4b67c446-4sm52\" (UID: \"93befab7-20fb-465f-97f0-b5fe1309113c\") " pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:54.828722 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.828687    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sm52"]
Jul 09 14:51:54.828965 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.828943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:54.862139 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.862107    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-rv4sx"]
Jul 09 14:51:54.865505 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.865488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:54.868552 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.868525    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 14:51:54.877050 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.877028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-rv4sx"]
Jul 09 14:51:54.910998 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.901553    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-rv4sx"]
Jul 09 14:51:54.911475 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:51:54.911443    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s4knm tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-s4knm tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-rv4sx" podUID="5b39e242-d112-4ddb-935e-a8aaf0fae263"
Jul 09 14:51:54.936971 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.936937    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b796b48cf-46pbq"]
Jul 09 14:51:54.940641 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.940617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:54.951159 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.951127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b796b48cf-46pbq"]
Jul 09 14:51:54.978798 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.978766    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sm52"]
Jul 09 14:51:54.981823 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:51:54.981791    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93befab7_20fb_465f_97f0_b5fe1309113c.slice/crio-a068308a1a6c126eb6199e8fc7cf56e3e5b3d70857ff9dc0ab09afc65efaae2e WatchSource:0}: Error finding container a068308a1a6c126eb6199e8fc7cf56e3e5b3d70857ff9dc0ab09afc65efaae2e: Status 404 returned error can't find the container with id a068308a1a6c126eb6199e8fc7cf56e3e5b3d70857ff9dc0ab09afc65efaae2e
Jul 09 14:51:54.982092 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.982074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4knm\" (UniqueName: \"kubernetes.io/projected/5b39e242-d112-4ddb-935e-a8aaf0fae263-kube-api-access-s4knm\") pod \"authorino-76877994-rv4sx\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") " pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:54.982171 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:54.982156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b39e242-d112-4ddb-935e-a8aaf0fae263-tls-cert\") pod \"authorino-76877994-rv4sx\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") " pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.071446 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.071420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.071446 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.071435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sm52" event={"ID":"93befab7-20fb-465f-97f0-b5fe1309113c","Type":"ContainerStarted","Data":"a068308a1a6c126eb6199e8fc7cf56e3e5b3d70857ff9dc0ab09afc65efaae2e"}
Jul 09 14:51:55.078508 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.078486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.083102 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.083085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4knm\" (UniqueName: \"kubernetes.io/projected/5b39e242-d112-4ddb-935e-a8aaf0fae263-kube-api-access-s4knm\") pod \"authorino-76877994-rv4sx\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") " pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.083171 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.083141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b39e242-d112-4ddb-935e-a8aaf0fae263-tls-cert\") pod \"authorino-76877994-rv4sx\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") " pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.083171 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.083163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-tls-cert\") pod \"authorino-6b796b48cf-46pbq\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") " pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.083251 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.083197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqb8z\" (UniqueName: \"kubernetes.io/projected/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-kube-api-access-zqb8z\") pod \"authorino-6b796b48cf-46pbq\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") " pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.085455 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.085434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b39e242-d112-4ddb-935e-a8aaf0fae263-tls-cert\") pod \"authorino-76877994-rv4sx\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") " pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.097386 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.097367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4knm\" (UniqueName: \"kubernetes.io/projected/5b39e242-d112-4ddb-935e-a8aaf0fae263-kube-api-access-s4knm\") pod \"authorino-76877994-rv4sx\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") " pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:55.183797 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.183766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4knm\" (UniqueName: \"kubernetes.io/projected/5b39e242-d112-4ddb-935e-a8aaf0fae263-kube-api-access-s4knm\") pod \"5b39e242-d112-4ddb-935e-a8aaf0fae263\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") "
Jul 09 14:51:55.183797 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.183800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b39e242-d112-4ddb-935e-a8aaf0fae263-tls-cert\") pod \"5b39e242-d112-4ddb-935e-a8aaf0fae263\" (UID: \"5b39e242-d112-4ddb-935e-a8aaf0fae263\") "
Jul 09 14:51:55.184053 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.183903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-tls-cert\") pod \"authorino-6b796b48cf-46pbq\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") " pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.184053 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.183954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqb8z\" (UniqueName: \"kubernetes.io/projected/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-kube-api-access-zqb8z\") pod \"authorino-6b796b48cf-46pbq\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") " pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.186479 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.186447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b39e242-d112-4ddb-935e-a8aaf0fae263-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5b39e242-d112-4ddb-935e-a8aaf0fae263" (UID: "5b39e242-d112-4ddb-935e-a8aaf0fae263"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:51:55.186591 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.186478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b39e242-d112-4ddb-935e-a8aaf0fae263-kube-api-access-s4knm" (OuterVolumeSpecName: "kube-api-access-s4knm") pod "5b39e242-d112-4ddb-935e-a8aaf0fae263" (UID: "5b39e242-d112-4ddb-935e-a8aaf0fae263"). InnerVolumeSpecName "kube-api-access-s4knm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:51:55.186591 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.186488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-tls-cert\") pod \"authorino-6b796b48cf-46pbq\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") " pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.192265 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.192246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqb8z\" (UniqueName: \"kubernetes.io/projected/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-kube-api-access-zqb8z\") pod \"authorino-6b796b48cf-46pbq\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") " pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.252362 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.252331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:51:55.285228 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.285200    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4knm\" (UniqueName: \"kubernetes.io/projected/5b39e242-d112-4ddb-935e-a8aaf0fae263-kube-api-access-s4knm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:55.285228 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.285227    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b39e242-d112-4ddb-935e-a8aaf0fae263-tls-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:55.382427 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:55.382396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b796b48cf-46pbq"]
Jul 09 14:51:55.384274 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:51:55.384249    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83d005d8_2f99_4925_ab0a_a2e0648d7dbb.slice/crio-54899e000dc8f69c1997c8e28f9d5f1f1bf6718427238bcb1246620cffb3fcec WatchSource:0}: Error finding container 54899e000dc8f69c1997c8e28f9d5f1f1bf6718427238bcb1246620cffb3fcec: Status 404 returned error can't find the container with id 54899e000dc8f69c1997c8e28f9d5f1f1bf6718427238bcb1246620cffb3fcec
Jul 09 14:51:56.077574 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.077469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b796b48cf-46pbq" event={"ID":"83d005d8-2f99-4925-ab0a-a2e0648d7dbb","Type":"ContainerStarted","Data":"c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234"}
Jul 09 14:51:56.077574 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.077521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b796b48cf-46pbq" event={"ID":"83d005d8-2f99-4925-ab0a-a2e0648d7dbb","Type":"ContainerStarted","Data":"54899e000dc8f69c1997c8e28f9d5f1f1bf6718427238bcb1246620cffb3fcec"}
Jul 09 14:51:56.078893 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.078874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rv4sx"
Jul 09 14:51:56.079241 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.078899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sm52" event={"ID":"93befab7-20fb-465f-97f0-b5fe1309113c","Type":"ContainerStarted","Data":"1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73"}
Jul 09 14:51:56.079241 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.078935    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-4sm52" podUID="93befab7-20fb-465f-97f0-b5fe1309113c" containerName="authorino" containerID="cri-o://1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73" gracePeriod=30
Jul 09 14:51:56.096507 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.096465    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b796b48cf-46pbq" podStartSLOduration=1.72693233 podStartE2EDuration="2.09645026s" totalImagesPullingTime="369.51793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:51:56.095688119 +0000 UTC m=+717.129216613" watchObservedRunningTime="2026-07-09 14:51:56.09645026 +0000 UTC m=+717.129978742"
Jul 09 14:51:56.132191 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.132150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-rv4sx"]
Jul 09 14:51:56.141515 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.141482    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-rv4sx"]
Jul 09 14:51:56.162700 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.162652    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-4sm52" podStartSLOduration=1.651685132 podStartE2EDuration="2.162634999s" totalImagesPullingTime="510.949867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:51:56.160768154 +0000 UTC m=+717.194296636" watchObservedRunningTime="2026-07-09 14:51:56.162634999 +0000 UTC m=+717.196163482"
Jul 09 14:51:56.330936 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.330858    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:56.394197 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.394169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz2j7\" (UniqueName: \"kubernetes.io/projected/93befab7-20fb-465f-97f0-b5fe1309113c-kube-api-access-vz2j7\") pod \"93befab7-20fb-465f-97f0-b5fe1309113c\" (UID: \"93befab7-20fb-465f-97f0-b5fe1309113c\") "
Jul 09 14:51:56.396330 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.396305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93befab7-20fb-465f-97f0-b5fe1309113c-kube-api-access-vz2j7" (OuterVolumeSpecName: "kube-api-access-vz2j7") pod "93befab7-20fb-465f-97f0-b5fe1309113c" (UID: "93befab7-20fb-465f-97f0-b5fe1309113c"). InnerVolumeSpecName "kube-api-access-vz2j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:51:56.494925 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:56.494892    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2j7\" (UniqueName: \"kubernetes.io/projected/93befab7-20fb-465f-97f0-b5fe1309113c-kube-api-access-vz2j7\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:51:57.085109 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.085071    2578 generic.go:356] "Generic (PLEG): container finished" podID="93befab7-20fb-465f-97f0-b5fe1309113c" containerID="1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73" exitCode=0
Jul 09 14:51:57.085515 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.085114    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4sm52"
Jul 09 14:51:57.085515 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.085112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sm52" event={"ID":"93befab7-20fb-465f-97f0-b5fe1309113c","Type":"ContainerDied","Data":"1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73"}
Jul 09 14:51:57.085515 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.085165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4sm52" event={"ID":"93befab7-20fb-465f-97f0-b5fe1309113c","Type":"ContainerDied","Data":"a068308a1a6c126eb6199e8fc7cf56e3e5b3d70857ff9dc0ab09afc65efaae2e"}
Jul 09 14:51:57.085515 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.085183    2578 scope.go:117] "RemoveContainer" containerID="1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73"
Jul 09 14:51:57.094515 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.094493    2578 scope.go:117] "RemoveContainer" containerID="1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73"
Jul 09 14:51:57.094762 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:51:57.094744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73\": container with ID starting with 1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73 not found: ID does not exist" containerID="1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73"
Jul 09 14:51:57.094826 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.094769    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73"} err="failed to get container status \"1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73\": rpc error: code = NotFound desc = could not find container \"1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73\": container with ID starting with 1ffd1ad467f0902309da63d21f90287efd73a6baa8301c19257ef148a496aa73 not found: ID does not exist"
Jul 09 14:51:57.107402 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.107375    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sm52"]
Jul 09 14:51:57.111426 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.111408    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4sm52"]
Jul 09 14:51:57.488742 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.488707    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b39e242-d112-4ddb-935e-a8aaf0fae263" path="/var/lib/kubelet/pods/5b39e242-d112-4ddb-935e-a8aaf0fae263/volumes"
Jul 09 14:51:57.488938 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:51:57.488925    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93befab7-20fb-465f-97f0-b5fe1309113c" path="/var/lib/kubelet/pods/93befab7-20fb-465f-97f0-b5fe1309113c/volumes"
Jul 09 14:52:00.472150 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.472114    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-fl9vs"]
Jul 09 14:52:00.472545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.472416    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93befab7-20fb-465f-97f0-b5fe1309113c" containerName="authorino"
Jul 09 14:52:00.472545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.472428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93befab7-20fb-465f-97f0-b5fe1309113c" containerName="authorino"
Jul 09 14:52:00.472545 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.472487    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93befab7-20fb-465f-97f0-b5fe1309113c" containerName="authorino"
Jul 09 14:52:00.478132 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.478107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.482243 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.482213    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 14:52:00.482243 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.482233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s9v2l\""
Jul 09 14:52:00.483776 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.483756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 14:52:00.492332 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.492307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-fl9vs"]
Jul 09 14:52:00.630343 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.630308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d40a248-e45c-4e53-9149-1ca17977197c-webhook-cert\") pod \"maas-controller-7bbd6859f5-fl9vs\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") " pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.630532 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.630358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwgx\" (UniqueName: \"kubernetes.io/projected/9d40a248-e45c-4e53-9149-1ca17977197c-kube-api-access-6dwgx\") pod \"maas-controller-7bbd6859f5-fl9vs\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") " pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.660882 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.660849    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-764f78bc44-rqqhh"]
Jul 09 14:52:00.664009 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.663993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.672909 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.672886    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764f78bc44-rqqhh"]
Jul 09 14:52:00.731256 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.731176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d40a248-e45c-4e53-9149-1ca17977197c-webhook-cert\") pod \"maas-controller-7bbd6859f5-fl9vs\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") " pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.731256 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.731230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwgx\" (UniqueName: \"kubernetes.io/projected/9d40a248-e45c-4e53-9149-1ca17977197c-kube-api-access-6dwgx\") pod \"maas-controller-7bbd6859f5-fl9vs\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") " pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.731477 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.731341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6qx\" (UniqueName: \"kubernetes.io/projected/fbc19388-efa2-4166-bc18-dc5f5d6f7639-kube-api-access-mk6qx\") pod \"maas-controller-764f78bc44-rqqhh\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") " pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.731477 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.731443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc19388-efa2-4166-bc18-dc5f5d6f7639-webhook-cert\") pod \"maas-controller-764f78bc44-rqqhh\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") " pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.733597 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.733580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d40a248-e45c-4e53-9149-1ca17977197c-webhook-cert\") pod \"maas-controller-7bbd6859f5-fl9vs\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") " pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.740824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.740801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwgx\" (UniqueName: \"kubernetes.io/projected/9d40a248-e45c-4e53-9149-1ca17977197c-kube-api-access-6dwgx\") pod \"maas-controller-7bbd6859f5-fl9vs\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") " pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.788993 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.788957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:00.831990 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.831925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc19388-efa2-4166-bc18-dc5f5d6f7639-webhook-cert\") pod \"maas-controller-764f78bc44-rqqhh\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") " pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.832162 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.832017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6qx\" (UniqueName: \"kubernetes.io/projected/fbc19388-efa2-4166-bc18-dc5f5d6f7639-kube-api-access-mk6qx\") pod \"maas-controller-764f78bc44-rqqhh\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") " pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.834397 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.834350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc19388-efa2-4166-bc18-dc5f5d6f7639-webhook-cert\") pod \"maas-controller-764f78bc44-rqqhh\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") " pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.849245 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.849211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6qx\" (UniqueName: \"kubernetes.io/projected/fbc19388-efa2-4166-bc18-dc5f5d6f7639-kube-api-access-mk6qx\") pod \"maas-controller-764f78bc44-rqqhh\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") " pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:00.924535 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.924512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-fl9vs"]
Jul 09 14:52:00.926579 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:52:00.926554    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d40a248_e45c_4e53_9149_1ca17977197c.slice/crio-0e1a545e2125c99ef2f306decbd8eaac58454da5ceec3ee7a22c40e0ec06dd8f WatchSource:0}: Error finding container 0e1a545e2125c99ef2f306decbd8eaac58454da5ceec3ee7a22c40e0ec06dd8f: Status 404 returned error can't find the container with id 0e1a545e2125c99ef2f306decbd8eaac58454da5ceec3ee7a22c40e0ec06dd8f
Jul 09 14:52:00.974594 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:00.974563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:01.105773 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:01.105734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs" event={"ID":"9d40a248-e45c-4e53-9149-1ca17977197c","Type":"ContainerStarted","Data":"0e1a545e2125c99ef2f306decbd8eaac58454da5ceec3ee7a22c40e0ec06dd8f"}
Jul 09 14:52:01.106468 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:01.106446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764f78bc44-rqqhh"]
Jul 09 14:52:01.107955 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:52:01.107933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc19388_efa2_4166_bc18_dc5f5d6f7639.slice/crio-f132f151a77edffd797b8629edeab014d4139073b562a8462439efc420dcd6d8 WatchSource:0}: Error finding container f132f151a77edffd797b8629edeab014d4139073b562a8462439efc420dcd6d8: Status 404 returned error can't find the container with id f132f151a77edffd797b8629edeab014d4139073b562a8462439efc420dcd6d8
Jul 09 14:52:02.117750 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:02.117671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764f78bc44-rqqhh" event={"ID":"fbc19388-efa2-4166-bc18-dc5f5d6f7639","Type":"ContainerStarted","Data":"f132f151a77edffd797b8629edeab014d4139073b562a8462439efc420dcd6d8"}
Jul 09 14:52:04.129771 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:04.129730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs" event={"ID":"9d40a248-e45c-4e53-9149-1ca17977197c","Type":"ContainerStarted","Data":"ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf"}
Jul 09 14:52:04.130234 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:04.129803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:04.131271 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:04.131236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764f78bc44-rqqhh" event={"ID":"fbc19388-efa2-4166-bc18-dc5f5d6f7639","Type":"ContainerStarted","Data":"c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb"}
Jul 09 14:52:04.131377 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:04.131295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:04.163772 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:04.163722    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs" podStartSLOduration=2.008571811 podStartE2EDuration="4.163704973s" totalImagesPullingTime="2.155133162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:04.162047529 +0000 UTC m=+725.195576029" watchObservedRunningTime="2026-07-09 14:52:04.163704973 +0000 UTC m=+725.197233500"
Jul 09 14:52:04.197758 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:04.197708    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-764f78bc44-rqqhh" podStartSLOduration=2.233277748 podStartE2EDuration="4.197691275s" totalImagesPullingTime="1.964413527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:04.196301611 +0000 UTC m=+725.229830113" watchObservedRunningTime="2026-07-09 14:52:04.197691275 +0000 UTC m=+725.231219769"
Jul 09 14:52:15.141066 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.141038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:52:15.141435 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.141103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:15.205002 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.204960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-fl9vs"]
Jul 09 14:52:15.205178 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.205151    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs" podUID="9d40a248-e45c-4e53-9149-1ca17977197c" containerName="manager" containerID="cri-o://ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf" gracePeriod=30
Jul 09 14:52:15.448363 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.448343    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:15.548707 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.548687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dwgx\" (UniqueName: \"kubernetes.io/projected/9d40a248-e45c-4e53-9149-1ca17977197c-kube-api-access-6dwgx\") pod \"9d40a248-e45c-4e53-9149-1ca17977197c\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") "
Jul 09 14:52:15.548951 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.548761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d40a248-e45c-4e53-9149-1ca17977197c-webhook-cert\") pod \"9d40a248-e45c-4e53-9149-1ca17977197c\" (UID: \"9d40a248-e45c-4e53-9149-1ca17977197c\") "
Jul 09 14:52:15.550724 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.550698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d40a248-e45c-4e53-9149-1ca17977197c-kube-api-access-6dwgx" (OuterVolumeSpecName: "kube-api-access-6dwgx") pod "9d40a248-e45c-4e53-9149-1ca17977197c" (UID: "9d40a248-e45c-4e53-9149-1ca17977197c"). InnerVolumeSpecName "kube-api-access-6dwgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:52:15.550829 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.550804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d40a248-e45c-4e53-9149-1ca17977197c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9d40a248-e45c-4e53-9149-1ca17977197c" (UID: "9d40a248-e45c-4e53-9149-1ca17977197c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:52:15.650152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.650130    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwgx\" (UniqueName: \"kubernetes.io/projected/9d40a248-e45c-4e53-9149-1ca17977197c-kube-api-access-6dwgx\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:52:15.650152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:15.650150    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d40a248-e45c-4e53-9149-1ca17977197c-webhook-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:52:16.183166 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.183133    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d40a248-e45c-4e53-9149-1ca17977197c" containerID="ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf" exitCode=0
Jul 09 14:52:16.183557 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.183173    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs"
Jul 09 14:52:16.183557 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.183185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs" event={"ID":"9d40a248-e45c-4e53-9149-1ca17977197c","Type":"ContainerDied","Data":"ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf"}
Jul 09 14:52:16.183557 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.183234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbd6859f5-fl9vs" event={"ID":"9d40a248-e45c-4e53-9149-1ca17977197c","Type":"ContainerDied","Data":"0e1a545e2125c99ef2f306decbd8eaac58454da5ceec3ee7a22c40e0ec06dd8f"}
Jul 09 14:52:16.183557 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.183257    2578 scope.go:117] "RemoveContainer" containerID="ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf"
Jul 09 14:52:16.192017 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.192000    2578 scope.go:117] "RemoveContainer" containerID="ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf"
Jul 09 14:52:16.192245 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:52:16.192229    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf\": container with ID starting with ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf not found: ID does not exist" containerID="ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf"
Jul 09 14:52:16.192285 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.192252    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf"} err="failed to get container status \"ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf\": rpc error: code = NotFound desc = could not find container \"ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf\": container with ID starting with ed276671abb1cbf7b9e010d21bd2b03fd62e5979ad833adcd01b40a65242cbaf not found: ID does not exist"
Jul 09 14:52:16.206871 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.206849    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-fl9vs"]
Jul 09 14:52:16.212937 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:16.212917    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbd6859f5-fl9vs"]
Jul 09 14:52:17.488144 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:17.488111    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d40a248-e45c-4e53-9149-1ca17977197c" path="/var/lib/kubelet/pods/9d40a248-e45c-4e53-9149-1ca17977197c/volumes"
Jul 09 14:52:34.669320 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.669266    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"]
Jul 09 14:52:34.669707 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.669637    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d40a248-e45c-4e53-9149-1ca17977197c" containerName="manager"
Jul 09 14:52:34.669707 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.669650    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d40a248-e45c-4e53-9149-1ca17977197c" containerName="manager"
Jul 09 14:52:34.669707 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.669696    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d40a248-e45c-4e53-9149-1ca17977197c" containerName="manager"
Jul 09 14:52:34.672149 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.672128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.676068 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.676048    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 09 14:52:34.676198 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.676179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wgbbv\""
Jul 09 14:52:34.677397 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.677378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwsw\" (UniqueName: \"kubernetes.io/projected/06e2e76e-391d-495c-b968-d507a689cc75-kube-api-access-5wwsw\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.677488 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.677430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e2e76e-391d-495c-b968-d507a689cc75-openshift-service-ca\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.677573 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.677554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e2e76e-391d-495c-b968-d507a689cc75-maas-api-tls\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.685169 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.685119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"]
Jul 09 14:52:34.778022 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.777973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e2e76e-391d-495c-b968-d507a689cc75-maas-api-tls\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.778142 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:52:34.778058    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 09 14:52:34.778142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.778070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwsw\" (UniqueName: \"kubernetes.io/projected/06e2e76e-391d-495c-b968-d507a689cc75-kube-api-access-5wwsw\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.778142 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:52:34.778105    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e2e76e-391d-495c-b968-d507a689cc75-maas-api-tls podName:06e2e76e-391d-495c-b968-d507a689cc75 nodeName:}" failed. No retries permitted until 2026-07-09 14:52:35.27809089 +0000 UTC m=+756.311619350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06e2e76e-391d-495c-b968-d507a689cc75-maas-api-tls") pod "maas-api-5d64f7d5c4-5tbm2" (UID: "06e2e76e-391d-495c-b968-d507a689cc75") : secret "maas-api-serving-cert" not found
Jul 09 14:52:34.778142 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.778131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e2e76e-391d-495c-b968-d507a689cc75-openshift-service-ca\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.778718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.778703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e2e76e-391d-495c-b968-d507a689cc75-openshift-service-ca\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:34.790232 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:34.790217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwsw\" (UniqueName: \"kubernetes.io/projected/06e2e76e-391d-495c-b968-d507a689cc75-kube-api-access-5wwsw\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:35.203051 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.203018    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-dlzgw"]
Jul 09 14:52:35.205167 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.205152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.208548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.208532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 09 14:52:35.208635 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.208619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tt4ds\""
Jul 09 14:52:35.217694 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.217671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-dlzgw"]
Jul 09 14:52:35.283558 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.283512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aea3be-e1be-4f86-8a40-61474409c4ab-plugins-config-volume\") pod \"payload-processing-58f86c7798-dlzgw\" (UID: \"b7aea3be-e1be-4f86-8a40-61474409c4ab\") " pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.283705 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.283584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e2e76e-391d-495c-b968-d507a689cc75-maas-api-tls\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:35.283705 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.283627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdlh\" (UniqueName: \"kubernetes.io/projected/b7aea3be-e1be-4f86-8a40-61474409c4ab-kube-api-access-2hdlh\") pod \"payload-processing-58f86c7798-dlzgw\" (UID: \"b7aea3be-e1be-4f86-8a40-61474409c4ab\") " pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.285951 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.285926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e2e76e-391d-495c-b968-d507a689cc75-maas-api-tls\") pod \"maas-api-5d64f7d5c4-5tbm2\" (UID: \"06e2e76e-391d-495c-b968-d507a689cc75\") " pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:35.384260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.384226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aea3be-e1be-4f86-8a40-61474409c4ab-plugins-config-volume\") pod \"payload-processing-58f86c7798-dlzgw\" (UID: \"b7aea3be-e1be-4f86-8a40-61474409c4ab\") " pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.384408 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.384371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdlh\" (UniqueName: \"kubernetes.io/projected/b7aea3be-e1be-4f86-8a40-61474409c4ab-kube-api-access-2hdlh\") pod \"payload-processing-58f86c7798-dlzgw\" (UID: \"b7aea3be-e1be-4f86-8a40-61474409c4ab\") " pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.384967 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.384941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7aea3be-e1be-4f86-8a40-61474409c4ab-plugins-config-volume\") pod \"payload-processing-58f86c7798-dlzgw\" (UID: \"b7aea3be-e1be-4f86-8a40-61474409c4ab\") " pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.393823 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.393801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdlh\" (UniqueName: \"kubernetes.io/projected/b7aea3be-e1be-4f86-8a40-61474409c4ab-kube-api-access-2hdlh\") pod \"payload-processing-58f86c7798-dlzgw\" (UID: \"b7aea3be-e1be-4f86-8a40-61474409c4ab\") " pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.516535 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.516429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:35.582810 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.582620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:35.654718 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.654695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-dlzgw"]
Jul 09 14:52:35.657130 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:52:35.657099    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aea3be_e1be_4f86_8a40_61474409c4ab.slice/crio-94ca2853792daac2945de5b481b390e1a96cce06c79dee08bda6d671879146cc WatchSource:0}: Error finding container 94ca2853792daac2945de5b481b390e1a96cce06c79dee08bda6d671879146cc: Status 404 returned error can't find the container with id 94ca2853792daac2945de5b481b390e1a96cce06c79dee08bda6d671879146cc
Jul 09 14:52:35.718150 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:35.718069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"]
Jul 09 14:52:35.721145 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:52:35.721118    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e2e76e_391d_495c_b968_d507a689cc75.slice/crio-7accadd32cc7304bb71ab907d6af7fced882d3da3419ae7c0d76a02b64462d18 WatchSource:0}: Error finding container 7accadd32cc7304bb71ab907d6af7fced882d3da3419ae7c0d76a02b64462d18: Status 404 returned error can't find the container with id 7accadd32cc7304bb71ab907d6af7fced882d3da3419ae7c0d76a02b64462d18
Jul 09 14:52:36.276850 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:36.276801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw" event={"ID":"b7aea3be-e1be-4f86-8a40-61474409c4ab","Type":"ContainerStarted","Data":"94ca2853792daac2945de5b481b390e1a96cce06c79dee08bda6d671879146cc"}
Jul 09 14:52:36.278292 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:36.278265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2" event={"ID":"06e2e76e-391d-495c-b968-d507a689cc75","Type":"ContainerStarted","Data":"7accadd32cc7304bb71ab907d6af7fced882d3da3419ae7c0d76a02b64462d18"}
Jul 09 14:52:38.289120 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:38.289084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2" event={"ID":"06e2e76e-391d-495c-b968-d507a689cc75","Type":"ContainerStarted","Data":"71282f1a04012157331b16220699f1d2ee145a977e35a7de67b05bec69de2445"}
Jul 09 14:52:38.289513 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:38.289151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:38.290509 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:38.290484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw" event={"ID":"b7aea3be-e1be-4f86-8a40-61474409c4ab","Type":"ContainerStarted","Data":"c9af006ee001dd34e4dbafeaf5282e35ce1a9e25222a8044eae8d980eda2a366"}
Jul 09 14:52:38.290640 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:38.290603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:52:38.307644 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:38.307603    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2" podStartSLOduration=2.742062971 podStartE2EDuration="4.307590866s" totalImagesPullingTime="1.565527895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:38.306261783 +0000 UTC m=+759.339790300" watchObservedRunningTime="2026-07-09 14:52:38.307590866 +0000 UTC m=+759.341119348"
Jul 09 14:52:38.323327 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:38.323286    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw" podStartSLOduration=1.719145487 podStartE2EDuration="3.323273054s" totalImagesPullingTime="1.604127567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:38.322599145 +0000 UTC m=+759.356127647" watchObservedRunningTime="2026-07-09 14:52:38.323273054 +0000 UTC m=+759.356801539"
Jul 09 14:52:44.300332 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:44.300302    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d64f7d5c4-5tbm2"
Jul 09 14:52:49.297522 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:52:49.297467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-dlzgw"
Jul 09 14:54:01.468646 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.468616    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"]
Jul 09 14:54:01.471903 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.471872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.474699 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.474677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 14:54:01.476204 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.476174    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 14:54:01.476308 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.476181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 09 14:54:01.476360 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.476345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cj4mh\""
Jul 09 14:54:01.489337 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.489316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"]
Jul 09 14:54:01.507084 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.507058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.507179 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.507090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvzx\" (UniqueName: \"kubernetes.io/projected/e47919ce-b31c-4b77-bcf6-dce037515912-kube-api-access-4rvzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.507179 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.507129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.507179 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.507160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47919ce-b31c-4b77-bcf6-dce037515912-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.507280 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.507190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.507280 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.507239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.607797 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.607770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.607924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.607808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47919ce-b31c-4b77-bcf6-dce037515912-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.607924 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.607899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.608090 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.607961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.608090 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.608051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.608090 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.608082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvzx\" (UniqueName: \"kubernetes.io/projected/e47919ce-b31c-4b77-bcf6-dce037515912-kube-api-access-4rvzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.608248 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.608178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.608302 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.608274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.608412 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.608388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.610115 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.610092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47919ce-b31c-4b77-bcf6-dce037515912-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.610437 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.610418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47919ce-b31c-4b77-bcf6-dce037515912-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.617275 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.617247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvzx\" (UniqueName: \"kubernetes.io/projected/e47919ce-b31c-4b77-bcf6-dce037515912-kube-api-access-4rvzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tj6d4\" (UID: \"e47919ce-b31c-4b77-bcf6-dce037515912\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.782582 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.782502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:01.913506 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:01.913477    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"]
Jul 09 14:54:01.914686 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:54:01.914658    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47919ce_b31c_4b77_bcf6_dce037515912.slice/crio-e9aeffc921b073d209cbd642f635c80fd9b7d32b28e4e851cf8fae1f5737a660 WatchSource:0}: Error finding container e9aeffc921b073d209cbd642f635c80fd9b7d32b28e4e851cf8fae1f5737a660: Status 404 returned error can't find the container with id e9aeffc921b073d209cbd642f635c80fd9b7d32b28e4e851cf8fae1f5737a660
Jul 09 14:54:02.676674 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:02.676640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4" event={"ID":"e47919ce-b31c-4b77-bcf6-dce037515912","Type":"ContainerStarted","Data":"e9aeffc921b073d209cbd642f635c80fd9b7d32b28e4e851cf8fae1f5737a660"}
Jul 09 14:54:03.683042 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:03.683006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4" event={"ID":"e47919ce-b31c-4b77-bcf6-dce037515912","Type":"ContainerStarted","Data":"c40b5b43f767ec53e0da179d016930486a3320b6f4f859ec939509245f6d0357"}
Jul 09 14:54:03.700863 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:03.700814    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4" podStartSLOduration=1.590507385 podStartE2EDuration="2.700797778s" totalImagesPullingTime="1.110290393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:54:03.699454366 +0000 UTC m=+844.732982868" watchObservedRunningTime="2026-07-09 14:54:03.700797778 +0000 UTC m=+844.734326280"
Jul 09 14:54:11.783112 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:11.783078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:11.783480 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:11.783143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:11.795256 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:11.795234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:12.733646 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:12.733620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tj6d4"
Jul 09 14:54:14.167485 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.167452    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"]
Jul 09 14:54:14.170928 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.170907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.174106 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.174081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 09 14:54:14.184024 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.184002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"]
Jul 09 14:54:14.194351 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.194325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.194442 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.194367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twjzl\" (UniqueName: \"kubernetes.io/projected/269b6d80-2780-40fe-9abf-85bfa43b5072-kube-api-access-twjzl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.194508 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.194490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.194549 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.194535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.194613 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.194595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b6d80-2780-40fe-9abf-85bfa43b5072-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.194672 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.194656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.295591 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.295561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.295591 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.295594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twjzl\" (UniqueName: \"kubernetes.io/projected/269b6d80-2780-40fe-9abf-85bfa43b5072-kube-api-access-twjzl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.295824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.295653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.295824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.295684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.295824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.295715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b6d80-2780-40fe-9abf-85bfa43b5072-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.295824 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.295754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.296053 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.296010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.296109 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.296067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.296153 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.296142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.297950 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.297922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b6d80-2780-40fe-9abf-85bfa43b5072-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.298315 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.298297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b6d80-2780-40fe-9abf-85bfa43b5072-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.304284 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.304262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twjzl\" (UniqueName: \"kubernetes.io/projected/269b6d80-2780-40fe-9abf-85bfa43b5072-kube-api-access-twjzl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt\" (UID: \"269b6d80-2780-40fe-9abf-85bfa43b5072\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.481850 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.481786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:14.614969 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.614944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"]
Jul 09 14:54:14.617573 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:54:14.617541    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269b6d80_2780_40fe_9abf_85bfa43b5072.slice/crio-8f6d2bff0d425c46a4cef650e5578378a35ef9999fe11d8f129986f755995d3b WatchSource:0}: Error finding container 8f6d2bff0d425c46a4cef650e5578378a35ef9999fe11d8f129986f755995d3b: Status 404 returned error can't find the container with id 8f6d2bff0d425c46a4cef650e5578378a35ef9999fe11d8f129986f755995d3b
Jul 09 14:54:14.731513 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:14.731481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt" event={"ID":"269b6d80-2780-40fe-9abf-85bfa43b5072","Type":"ContainerStarted","Data":"8f6d2bff0d425c46a4cef650e5578378a35ef9999fe11d8f129986f755995d3b"}
Jul 09 14:54:15.737235 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:15.737192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt" event={"ID":"269b6d80-2780-40fe-9abf-85bfa43b5072","Type":"ContainerStarted","Data":"81e09afcaa31b8480eb5422f4cc490c4ddaf4111c376daf9ea7ccba0b3f076e9"}
Jul 09 14:54:15.757836 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:15.757778    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt" podStartSLOduration=1.498772909 podStartE2EDuration="1.757763119s" totalImagesPullingTime="258.99021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:54:15.756871621 +0000 UTC m=+856.790400128" watchObservedRunningTime="2026-07-09 14:54:15.757763119 +0000 UTC m=+856.791291600"
Jul 09 14:54:24.482205 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:24.482169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:24.482598 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:24.482238    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:24.494432 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:24.494409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:24.788005 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:24.787920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt"
Jul 09 14:54:59.359152 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:59.359121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:54:59.361103 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:54:59.361070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:56:01.824721 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:01.824688    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b796b48cf-46pbq"]
Jul 09 14:56:01.825314 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:01.824899    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b796b48cf-46pbq" podUID="83d005d8-2f99-4925-ab0a-a2e0648d7dbb" containerName="authorino" containerID="cri-o://c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234" gracePeriod=30
Jul 09 14:56:02.072492 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.072462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:56:02.204752 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.204721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-tls-cert\") pod \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") "
Jul 09 14:56:02.204911 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.204778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqb8z\" (UniqueName: \"kubernetes.io/projected/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-kube-api-access-zqb8z\") pod \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\" (UID: \"83d005d8-2f99-4925-ab0a-a2e0648d7dbb\") "
Jul 09 14:56:02.206667 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.206643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-kube-api-access-zqb8z" (OuterVolumeSpecName: "kube-api-access-zqb8z") pod "83d005d8-2f99-4925-ab0a-a2e0648d7dbb" (UID: "83d005d8-2f99-4925-ab0a-a2e0648d7dbb"). InnerVolumeSpecName "kube-api-access-zqb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:56:02.216472 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.216445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "83d005d8-2f99-4925-ab0a-a2e0648d7dbb" (UID: "83d005d8-2f99-4925-ab0a-a2e0648d7dbb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:56:02.224774 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.224750    2578 generic.go:356] "Generic (PLEG): container finished" podID="83d005d8-2f99-4925-ab0a-a2e0648d7dbb" containerID="c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234" exitCode=0
Jul 09 14:56:02.224869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.224793    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b796b48cf-46pbq"
Jul 09 14:56:02.224869 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.224830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b796b48cf-46pbq" event={"ID":"83d005d8-2f99-4925-ab0a-a2e0648d7dbb","Type":"ContainerDied","Data":"c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234"}
Jul 09 14:56:02.224952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.224873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b796b48cf-46pbq" event={"ID":"83d005d8-2f99-4925-ab0a-a2e0648d7dbb","Type":"ContainerDied","Data":"54899e000dc8f69c1997c8e28f9d5f1f1bf6718427238bcb1246620cffb3fcec"}
Jul 09 14:56:02.224952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.224891    2578 scope.go:117] "RemoveContainer" containerID="c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234"
Jul 09 14:56:02.233316 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.233302    2578 scope.go:117] "RemoveContainer" containerID="c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234"
Jul 09 14:56:02.233555 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:56:02.233540    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234\": container with ID starting with c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234 not found: ID does not exist" containerID="c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234"
Jul 09 14:56:02.233603 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.233563    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234"} err="failed to get container status \"c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234\": rpc error: code = NotFound desc = could not find container \"c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234\": container with ID starting with c8d1fa1371a76e267c47e5a88530992f98539b3813f294dbaae899d868d07234 not found: ID does not exist"
Jul 09 14:56:02.249373 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.249349    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b796b48cf-46pbq"]
Jul 09 14:56:02.253478 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.253459    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b796b48cf-46pbq"]
Jul 09 14:56:02.305477 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.305457    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-tls-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:56:02.305477 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:02.305476    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqb8z\" (UniqueName: \"kubernetes.io/projected/83d005d8-2f99-4925-ab0a-a2e0648d7dbb-kube-api-access-zqb8z\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:56:03.489772 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:56:03.489737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d005d8-2f99-4925-ab0a-a2e0648d7dbb" path="/var/lib/kubelet/pods/83d005d8-2f99-4925-ab0a-a2e0648d7dbb/volumes"
Jul 09 14:57:26.996275 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:26.996243    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-764f78bc44-rqqhh"]
Jul 09 14:57:26.996799 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:26.996474    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-764f78bc44-rqqhh" podUID="fbc19388-efa2-4166-bc18-dc5f5d6f7639" containerName="manager" containerID="cri-o://c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb" gracePeriod=30
Jul 09 14:57:27.243991 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.243960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:57:27.352034 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.351950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc19388-efa2-4166-bc18-dc5f5d6f7639-webhook-cert\") pod \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") "
Jul 09 14:57:27.352169 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.352039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk6qx\" (UniqueName: \"kubernetes.io/projected/fbc19388-efa2-4166-bc18-dc5f5d6f7639-kube-api-access-mk6qx\") pod \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\" (UID: \"fbc19388-efa2-4166-bc18-dc5f5d6f7639\") "
Jul 09 14:57:27.353947 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.353918    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc19388-efa2-4166-bc18-dc5f5d6f7639-kube-api-access-mk6qx" (OuterVolumeSpecName: "kube-api-access-mk6qx") pod "fbc19388-efa2-4166-bc18-dc5f5d6f7639" (UID: "fbc19388-efa2-4166-bc18-dc5f5d6f7639"). InnerVolumeSpecName "kube-api-access-mk6qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 14:57:27.354073 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.353967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc19388-efa2-4166-bc18-dc5f5d6f7639-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fbc19388-efa2-4166-bc18-dc5f5d6f7639" (UID: "fbc19388-efa2-4166-bc18-dc5f5d6f7639"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 14:57:27.453151 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.453121    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc19388-efa2-4166-bc18-dc5f5d6f7639-webhook-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:57:27.453151 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.453144    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk6qx\" (UniqueName: \"kubernetes.io/projected/fbc19388-efa2-4166-bc18-dc5f5d6f7639-kube-api-access-mk6qx\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 14:57:27.601729 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.601698    2578 generic.go:356] "Generic (PLEG): container finished" podID="fbc19388-efa2-4166-bc18-dc5f5d6f7639" containerID="c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb" exitCode=0
Jul 09 14:57:27.601889 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.601755    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764f78bc44-rqqhh"
Jul 09 14:57:27.601889 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.601782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764f78bc44-rqqhh" event={"ID":"fbc19388-efa2-4166-bc18-dc5f5d6f7639","Type":"ContainerDied","Data":"c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb"}
Jul 09 14:57:27.601889 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.601832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764f78bc44-rqqhh" event={"ID":"fbc19388-efa2-4166-bc18-dc5f5d6f7639","Type":"ContainerDied","Data":"f132f151a77edffd797b8629edeab014d4139073b562a8462439efc420dcd6d8"}
Jul 09 14:57:27.601889 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.601855    2578 scope.go:117] "RemoveContainer" containerID="c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb"
Jul 09 14:57:27.610610 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.610593    2578 scope.go:117] "RemoveContainer" containerID="c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb"
Jul 09 14:57:27.610953 ip-10-0-142-236 kubenswrapper[2578]: E0709 14:57:27.610927    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb\": container with ID starting with c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb not found: ID does not exist" containerID="c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb"
Jul 09 14:57:27.611049 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.610964    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb"} err="failed to get container status \"c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb\": rpc error: code = NotFound desc = could not find container \"c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb\": container with ID starting with c3a22529e4e079233034ebfa2f311fedfb7aabca4324c824bb025ebfe45bd9bb not found: ID does not exist"
Jul 09 14:57:27.619548 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.619527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-764f78bc44-rqqhh"]
Jul 09 14:57:27.622104 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:27.622083    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-764f78bc44-rqqhh"]
Jul 09 14:57:28.397588 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.397551    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-764f78bc44-jg4b2"]
Jul 09 14:57:28.397952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.397899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d005d8-2f99-4925-ab0a-a2e0648d7dbb" containerName="authorino"
Jul 09 14:57:28.397952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.397911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d005d8-2f99-4925-ab0a-a2e0648d7dbb" containerName="authorino"
Jul 09 14:57:28.397952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.397929    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc19388-efa2-4166-bc18-dc5f5d6f7639" containerName="manager"
Jul 09 14:57:28.397952 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.397934    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc19388-efa2-4166-bc18-dc5f5d6f7639" containerName="manager"
Jul 09 14:57:28.398101 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.398030    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc19388-efa2-4166-bc18-dc5f5d6f7639" containerName="manager"
Jul 09 14:57:28.398101 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.398045    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d005d8-2f99-4925-ab0a-a2e0648d7dbb" containerName="authorino"
Jul 09 14:57:28.402260 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.402244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.405182 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.405152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 14:57:28.405329 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.405159    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s9v2l\""
Jul 09 14:57:28.406761 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.406580    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 14:57:28.408765 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.408745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764f78bc44-jg4b2"]
Jul 09 14:57:28.463020 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.462996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbqr\" (UniqueName: \"kubernetes.io/projected/9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0-kube-api-access-7xbqr\") pod \"maas-controller-764f78bc44-jg4b2\" (UID: \"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0\") " pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.463145 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.463032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0-webhook-cert\") pod \"maas-controller-764f78bc44-jg4b2\" (UID: \"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0\") " pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.564031 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.564007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbqr\" (UniqueName: \"kubernetes.io/projected/9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0-kube-api-access-7xbqr\") pod \"maas-controller-764f78bc44-jg4b2\" (UID: \"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0\") " pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.564161 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.564049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0-webhook-cert\") pod \"maas-controller-764f78bc44-jg4b2\" (UID: \"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0\") " pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.566408 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.566384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0-webhook-cert\") pod \"maas-controller-764f78bc44-jg4b2\" (UID: \"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0\") " pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.572766 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.572742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbqr\" (UniqueName: \"kubernetes.io/projected/9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0-kube-api-access-7xbqr\") pod \"maas-controller-764f78bc44-jg4b2\" (UID: \"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0\") " pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.713452 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.713424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:28.892846 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.892822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764f78bc44-jg4b2"]
Jul 09 14:57:28.895494 ip-10-0-142-236 kubenswrapper[2578]: W0709 14:57:28.895468    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae0ebb4_4b89_479c_8f62_d0677d1a9ff0.slice/crio-d54c3a1812d2e047ff0fa4b40382fb8bc51709f08a1b78e2231b93a61489e7c3 WatchSource:0}: Error finding container d54c3a1812d2e047ff0fa4b40382fb8bc51709f08a1b78e2231b93a61489e7c3: Status 404 returned error can't find the container with id d54c3a1812d2e047ff0fa4b40382fb8bc51709f08a1b78e2231b93a61489e7c3
Jul 09 14:57:28.896745 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:28.896725    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 14:57:29.495926 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:29.495890    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc19388-efa2-4166-bc18-dc5f5d6f7639" path="/var/lib/kubelet/pods/fbc19388-efa2-4166-bc18-dc5f5d6f7639/volumes"
Jul 09 14:57:29.614486 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:29.614447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764f78bc44-jg4b2" event={"ID":"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0","Type":"ContainerStarted","Data":"03069019753ce163a673574def67ceff0453f9804dda2c5ba2d38d70f5cf7955"}
Jul 09 14:57:29.614629 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:29.614512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764f78bc44-jg4b2" event={"ID":"9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0","Type":"ContainerStarted","Data":"d54c3a1812d2e047ff0fa4b40382fb8bc51709f08a1b78e2231b93a61489e7c3"}
Jul 09 14:57:29.614629 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:29.614539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:57:29.633791 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:29.633747    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-764f78bc44-jg4b2" podStartSLOduration=1.107386876 podStartE2EDuration="1.633732823s" totalImagesPullingTime="526.345947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:57:29.63120131 +0000 UTC m=+1050.664729812" watchObservedRunningTime="2026-07-09 14:57:29.633732823 +0000 UTC m=+1050.667261305"
Jul 09 14:57:40.624530 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:57:40.624504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-764f78bc44-jg4b2"
Jul 09 14:59:59.384620 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:59:59.384587    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 14:59:59.387449 ip-10-0-142-236 kubenswrapper[2578]: I0709 14:59:59.387427    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:00:00.145363 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.145331    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"]
Jul 09 15:00:00.148422 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.148404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:00:00.172333 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.172307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"]
Jul 09 15:00:00.201536 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.201511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2z9\" (UniqueName: \"kubernetes.io/projected/634f81dc-92d8-4a0e-9973-6a28e46b594d-kube-api-access-dz2z9\") pod \"maas-api-key-cleanup-29726820-q6b75\" (UID: \"634f81dc-92d8-4a0e-9973-6a28e46b594d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:00:00.302371 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.302341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2z9\" (UniqueName: \"kubernetes.io/projected/634f81dc-92d8-4a0e-9973-6a28e46b594d-kube-api-access-dz2z9\") pod \"maas-api-key-cleanup-29726820-q6b75\" (UID: \"634f81dc-92d8-4a0e-9973-6a28e46b594d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:00:00.311949 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.311922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2z9\" (UniqueName: \"kubernetes.io/projected/634f81dc-92d8-4a0e-9973-6a28e46b594d-kube-api-access-dz2z9\") pod \"maas-api-key-cleanup-29726820-q6b75\" (UID: \"634f81dc-92d8-4a0e-9973-6a28e46b594d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:00:00.458049 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.458026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:00:00.585590 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:00.585569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"]
Jul 09 15:00:00.587850 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:00:00.587824    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634f81dc_92d8_4a0e_9973_6a28e46b594d.slice/crio-999a3a1343bae2d94b3d11ac4d7d3d51360c603dce27b87421d7ded623dfcb0a WatchSource:0}: Error finding container 999a3a1343bae2d94b3d11ac4d7d3d51360c603dce27b87421d7ded623dfcb0a: Status 404 returned error can't find the container with id 999a3a1343bae2d94b3d11ac4d7d3d51360c603dce27b87421d7ded623dfcb0a
Jul 09 15:00:01.284549 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:01.284516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75" event={"ID":"634f81dc-92d8-4a0e-9973-6a28e46b594d","Type":"ContainerStarted","Data":"999a3a1343bae2d94b3d11ac4d7d3d51360c603dce27b87421d7ded623dfcb0a"}
Jul 09 15:00:19.368722 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:19.368640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75" event={"ID":"634f81dc-92d8-4a0e-9973-6a28e46b594d","Type":"ContainerStarted","Data":"74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6"}
Jul 09 15:00:19.388328 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:00:19.388287    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75" podStartSLOduration=0.931534186 podStartE2EDuration="19.388272965s" totalImagesPullingTime="18.456738779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:00:19.386684652 +0000 UTC m=+1220.420213159" watchObservedRunningTime="2026-07-09 15:00:19.388272965 +0000 UTC m=+1220.421801446"
Jul 09 15:01:00.018830 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:00.018747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"]
Jul 09 15:01:00.021430 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:00.018950    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75" podUID="634f81dc-92d8-4a0e-9973-6a28e46b594d" containerName="cleanup" containerID="cri-o://74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6" gracePeriod=30
Jul 09 15:01:30.198455 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.198426    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:01:30.285476 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.285378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz2z9\" (UniqueName: \"kubernetes.io/projected/634f81dc-92d8-4a0e-9973-6a28e46b594d-kube-api-access-dz2z9\") pod \"634f81dc-92d8-4a0e-9973-6a28e46b594d\" (UID: \"634f81dc-92d8-4a0e-9973-6a28e46b594d\") "
Jul 09 15:01:30.287478 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.287447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634f81dc-92d8-4a0e-9973-6a28e46b594d-kube-api-access-dz2z9" (OuterVolumeSpecName: "kube-api-access-dz2z9") pod "634f81dc-92d8-4a0e-9973-6a28e46b594d" (UID: "634f81dc-92d8-4a0e-9973-6a28e46b594d"). InnerVolumeSpecName "kube-api-access-dz2z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:01:30.386512 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.386485    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2z9\" (UniqueName: \"kubernetes.io/projected/634f81dc-92d8-4a0e-9973-6a28e46b594d-kube-api-access-dz2z9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:01:30.679031 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.678994    2578 generic.go:356] "Generic (PLEG): container finished" podID="634f81dc-92d8-4a0e-9973-6a28e46b594d" containerID="74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6" exitCode=137
Jul 09 15:01:30.679031 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.679033    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"
Jul 09 15:01:30.679298 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.679066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75" event={"ID":"634f81dc-92d8-4a0e-9973-6a28e46b594d","Type":"ContainerDied","Data":"74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6"}
Jul 09 15:01:30.679298 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.679118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75" event={"ID":"634f81dc-92d8-4a0e-9973-6a28e46b594d","Type":"ContainerDied","Data":"999a3a1343bae2d94b3d11ac4d7d3d51360c603dce27b87421d7ded623dfcb0a"}
Jul 09 15:01:30.679298 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.679139    2578 scope.go:117] "RemoveContainer" containerID="74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6"
Jul 09 15:01:30.687730 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.687717    2578 scope.go:117] "RemoveContainer" containerID="74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6"
Jul 09 15:01:30.687971 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:01:30.687953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6\": container with ID starting with 74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6 not found: ID does not exist" containerID="74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6"
Jul 09 15:01:30.688048 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.687993    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6"} err="failed to get container status \"74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6\": rpc error: code = NotFound desc = could not find container \"74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6\": container with ID starting with 74113927c4bb5417e4b1d426e635c09bd3956053456217b24581a73fc5a353c6 not found: ID does not exist"
Jul 09 15:01:30.702430 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.702409    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"]
Jul 09 15:01:30.705161 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:30.705141    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726820-q6b75"]
Jul 09 15:01:31.488816 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:01:31.488783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634f81dc-92d8-4a0e-9973-6a28e46b594d" path="/var/lib/kubelet/pods/634f81dc-92d8-4a0e-9973-6a28e46b594d/volumes"
Jul 09 15:02:19.160420 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:19.160385    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"]
Jul 09 15:02:19.160895 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:19.160567    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2" podUID="ead4ca74-7218-4116-ab54-a6fb55bd49e6" containerName="manager" containerID="cri-o://08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc" gracePeriod=10
Jul 09 15:02:22.317460 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.317433    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 15:02:22.390616 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.390551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6f9x\" (UniqueName: \"kubernetes.io/projected/ead4ca74-7218-4116-ab54-a6fb55bd49e6-kube-api-access-n6f9x\") pod \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") "
Jul 09 15:02:22.390741 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.390642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead4ca74-7218-4116-ab54-a6fb55bd49e6-extensions-socket-volume\") pod \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\" (UID: \"ead4ca74-7218-4116-ab54-a6fb55bd49e6\") "
Jul 09 15:02:22.391046 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.391023    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead4ca74-7218-4116-ab54-a6fb55bd49e6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ead4ca74-7218-4116-ab54-a6fb55bd49e6" (UID: "ead4ca74-7218-4116-ab54-a6fb55bd49e6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:02:22.392559 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.392537    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead4ca74-7218-4116-ab54-a6fb55bd49e6-kube-api-access-n6f9x" (OuterVolumeSpecName: "kube-api-access-n6f9x") pod "ead4ca74-7218-4116-ab54-a6fb55bd49e6" (UID: "ead4ca74-7218-4116-ab54-a6fb55bd49e6"). InnerVolumeSpecName "kube-api-access-n6f9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:02:22.491341 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.491316    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead4ca74-7218-4116-ab54-a6fb55bd49e6-extensions-socket-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:02:22.491341 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.491339    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6f9x\" (UniqueName: \"kubernetes.io/projected/ead4ca74-7218-4116-ab54-a6fb55bd49e6-kube-api-access-n6f9x\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:02:22.914781 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.914744    2578 generic.go:356] "Generic (PLEG): container finished" podID="ead4ca74-7218-4116-ab54-a6fb55bd49e6" containerID="08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc" exitCode=0
Jul 09 15:02:22.914992 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.914809    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"
Jul 09 15:02:22.914992 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.914834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2" event={"ID":"ead4ca74-7218-4116-ab54-a6fb55bd49e6","Type":"ContainerDied","Data":"08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc"}
Jul 09 15:02:22.914992 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.914922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2" event={"ID":"ead4ca74-7218-4116-ab54-a6fb55bd49e6","Type":"ContainerDied","Data":"ba9567ad6497685f313810619a92b9eb567cd5a6eff8eb603bbefe033ee1a771"}
Jul 09 15:02:22.914992 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.914944    2578 scope.go:117] "RemoveContainer" containerID="08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc"
Jul 09 15:02:22.924685 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.924669    2578 scope.go:117] "RemoveContainer" containerID="08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc"
Jul 09 15:02:22.924931 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:02:22.924912    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc\": container with ID starting with 08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc not found: ID does not exist" containerID="08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc"
Jul 09 15:02:22.925001 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.924941    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc"} err="failed to get container status \"08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc\": rpc error: code = NotFound desc = could not find container \"08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc\": container with ID starting with 08be62438c64d407e5c3f72219b815d5c591406fa75011880532b1e7f6001ccc not found: ID does not exist"
Jul 09 15:02:22.937962 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.937943    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"]
Jul 09 15:02:22.942565 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:22.942544    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-75jx2"]
Jul 09 15:02:23.489134 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:02:23.489106    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead4ca74-7218-4116-ab54-a6fb55bd49e6" path="/var/lib/kubelet/pods/ead4ca74-7218-4116-ab54-a6fb55bd49e6/volumes"
Jul 09 15:03:20.360146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360114    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"]
Jul 09 15:03:20.360570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360449    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634f81dc-92d8-4a0e-9973-6a28e46b594d" containerName="cleanup"
Jul 09 15:03:20.360570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="634f81dc-92d8-4a0e-9973-6a28e46b594d" containerName="cleanup"
Jul 09 15:03:20.360570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360472    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead4ca74-7218-4116-ab54-a6fb55bd49e6" containerName="manager"
Jul 09 15:03:20.360570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead4ca74-7218-4116-ab54-a6fb55bd49e6" containerName="manager"
Jul 09 15:03:20.360570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360545    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead4ca74-7218-4116-ab54-a6fb55bd49e6" containerName="manager"
Jul 09 15:03:20.360570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.360557    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="634f81dc-92d8-4a0e-9973-6a28e46b594d" containerName="cleanup"
Jul 09 15:03:20.364633 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.364616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.368648 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.368621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gf4fj\""
Jul 09 15:03:20.385022 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.385002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"]
Jul 09 15:03:20.442831 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.442802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkxd\" (UniqueName: \"kubernetes.io/projected/d9806c78-3080-4ec4-8257-98adb745853a-kube-api-access-8nkxd\") pod \"kuadrant-operator-controller-manager-86878c769f-6g2rv\" (UID: \"d9806c78-3080-4ec4-8257-98adb745853a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.442946 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.442875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9806c78-3080-4ec4-8257-98adb745853a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6g2rv\" (UID: \"d9806c78-3080-4ec4-8257-98adb745853a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.543213 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.543187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9806c78-3080-4ec4-8257-98adb745853a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6g2rv\" (UID: \"d9806c78-3080-4ec4-8257-98adb745853a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.543299 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.543234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkxd\" (UniqueName: \"kubernetes.io/projected/d9806c78-3080-4ec4-8257-98adb745853a-kube-api-access-8nkxd\") pod \"kuadrant-operator-controller-manager-86878c769f-6g2rv\" (UID: \"d9806c78-3080-4ec4-8257-98adb745853a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.543629 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.543612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9806c78-3080-4ec4-8257-98adb745853a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6g2rv\" (UID: \"d9806c78-3080-4ec4-8257-98adb745853a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.553705 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.553680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkxd\" (UniqueName: \"kubernetes.io/projected/d9806c78-3080-4ec4-8257-98adb745853a-kube-api-access-8nkxd\") pod \"kuadrant-operator-controller-manager-86878c769f-6g2rv\" (UID: \"d9806c78-3080-4ec4-8257-98adb745853a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.674411 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.674391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:20.810160 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:20.810126    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"]
Jul 09 15:03:20.816111 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:03:20.816084    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9806c78_3080_4ec4_8257_98adb745853a.slice/crio-4820dc03d7ae5fb3dea5e013e950729c058dac3b9df4a152dc41f92fd7647619 WatchSource:0}: Error finding container 4820dc03d7ae5fb3dea5e013e950729c058dac3b9df4a152dc41f92fd7647619: Status 404 returned error can't find the container with id 4820dc03d7ae5fb3dea5e013e950729c058dac3b9df4a152dc41f92fd7647619
Jul 09 15:03:21.175955 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:21.175922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv" event={"ID":"d9806c78-3080-4ec4-8257-98adb745853a","Type":"ContainerStarted","Data":"50a14da1b98e20c1b2be51afd1f4a957a7a3159ae702ec1e65c6ccd7920b5fff"}
Jul 09 15:03:21.176129 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:21.175993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv" event={"ID":"d9806c78-3080-4ec4-8257-98adb745853a","Type":"ContainerStarted","Data":"4820dc03d7ae5fb3dea5e013e950729c058dac3b9df4a152dc41f92fd7647619"}
Jul 09 15:03:21.176129 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:21.176022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:03:21.206608 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:21.206557    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv" podStartSLOduration=1.206541431 podStartE2EDuration="1.206541431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:03:21.203880247 +0000 UTC m=+1402.237408729" watchObservedRunningTime="2026-07-09 15:03:21.206541431 +0000 UTC m=+1402.240069914"
Jul 09 15:03:32.181859 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:03:32.181831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6g2rv"
Jul 09 15:04:59.409619 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:04:59.409588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:04:59.414051 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:04:59.414031    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:09:09.964311 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:09.964271    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"]
Jul 09 15:09:09.967632 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:09.967613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:09.970642 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:09.970620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4d23c1d4-serving-cert\""
Jul 09 15:09:09.977823 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:09.977799    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"]
Jul 09 15:09:10.126020 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.125958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-openshift-service-ca\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.126171 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.126046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.126171 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.126159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbpq\" (UniqueName: \"kubernetes.io/projected/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-kube-api-access-7gbpq\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.226852 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.226781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.227026 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.226868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbpq\" (UniqueName: \"kubernetes.io/projected/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-kube-api-access-7gbpq\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.227026 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.226901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-openshift-service-ca\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.227026 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:10.226927    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-serving-cert: secret "maas-api-e2e-ait-4d23c1d4-serving-cert" not found
Jul 09 15:09:10.227218 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:10.227035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls podName:15ace234-9ddb-4fe5-ad6d-7a89587b51bb nodeName:}" failed. No retries permitted until 2026-07-09 15:09:10.727015498 +0000 UTC m=+1751.760543966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls") pod "maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" (UID: "15ace234-9ddb-4fe5-ad6d-7a89587b51bb") : secret "maas-api-e2e-ait-4d23c1d4-serving-cert" not found
Jul 09 15:09:10.227576 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.227554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-openshift-service-ca\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.236358 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.236330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbpq\" (UniqueName: \"kubernetes.io/projected/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-kube-api-access-7gbpq\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.731634 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.731606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.734052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.734028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls\") pod \"maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:10.878508 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:10.878480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:11.015874 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:11.015804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"]
Jul 09 15:09:11.019912 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:09:11.019822    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ace234_9ddb_4fe5_ad6d_7a89587b51bb.slice/crio-d30c2b69565303bfd1920196e695f76f7c5a1bc7a59457c40187d0443ffd212b WatchSource:0}: Error finding container d30c2b69565303bfd1920196e695f76f7c5a1bc7a59457c40187d0443ffd212b: Status 404 returned error can't find the container with id d30c2b69565303bfd1920196e695f76f7c5a1bc7a59457c40187d0443ffd212b
Jul 09 15:09:11.025390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:11.025369    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 15:09:11.758499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:11.758425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" event={"ID":"15ace234-9ddb-4fe5-ad6d-7a89587b51bb","Type":"ContainerStarted","Data":"70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4"}
Jul 09 15:09:11.758499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:11.758471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" event={"ID":"15ace234-9ddb-4fe5-ad6d-7a89587b51bb","Type":"ContainerStarted","Data":"d30c2b69565303bfd1920196e695f76f7c5a1bc7a59457c40187d0443ffd212b"}
Jul 09 15:09:11.758682 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:11.758528    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:11.776792 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:11.776751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" podStartSLOduration=2.323506425 podStartE2EDuration="2.776737196s" totalImagesPullingTime="453.230771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:09:11.775389868 +0000 UTC m=+1752.808918351" watchObservedRunningTime="2026-07-09 15:09:11.776737196 +0000 UTC m=+1752.810265679"
Jul 09 15:09:15.448005 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.447957    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"]
Jul 09 15:09:15.448465 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.448161    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" podUID="15ace234-9ddb-4fe5-ad6d-7a89587b51bb" containerName="maas-api" containerID="cri-o://70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4" gracePeriod=30
Jul 09 15:09:15.695495 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.695455    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:15.768125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.768069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gbpq\" (UniqueName: \"kubernetes.io/projected/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-kube-api-access-7gbpq\") pod \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") "
Jul 09 15:09:15.768125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.768110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls\") pod \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") "
Jul 09 15:09:15.768308 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.768162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-openshift-service-ca\") pod \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\" (UID: \"15ace234-9ddb-4fe5-ad6d-7a89587b51bb\") "
Jul 09 15:09:15.768520 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.768498    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15ace234-9ddb-4fe5-ad6d-7a89587b51bb" (UID: "15ace234-9ddb-4fe5-ad6d-7a89587b51bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:09:15.770151 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.770116    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-kube-api-access-7gbpq" (OuterVolumeSpecName: "kube-api-access-7gbpq") pod "15ace234-9ddb-4fe5-ad6d-7a89587b51bb" (UID: "15ace234-9ddb-4fe5-ad6d-7a89587b51bb"). InnerVolumeSpecName "kube-api-access-7gbpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:09:15.770240 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.770215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15ace234-9ddb-4fe5-ad6d-7a89587b51bb" (UID: "15ace234-9ddb-4fe5-ad6d-7a89587b51bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:09:15.777220 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.777194    2578 generic.go:356] "Generic (PLEG): container finished" podID="15ace234-9ddb-4fe5-ad6d-7a89587b51bb" containerID="70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4" exitCode=0
Jul 09 15:09:15.777338 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.777246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"
Jul 09 15:09:15.777338 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.777256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" event={"ID":"15ace234-9ddb-4fe5-ad6d-7a89587b51bb","Type":"ContainerDied","Data":"70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4"}
Jul 09 15:09:15.777338 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.777307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr" event={"ID":"15ace234-9ddb-4fe5-ad6d-7a89587b51bb","Type":"ContainerDied","Data":"d30c2b69565303bfd1920196e695f76f7c5a1bc7a59457c40187d0443ffd212b"}
Jul 09 15:09:15.777492 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.777341    2578 scope.go:117] "RemoveContainer" containerID="70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4"
Jul 09 15:09:15.785805 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.785785    2578 scope.go:117] "RemoveContainer" containerID="70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4"
Jul 09 15:09:15.786057 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:15.786041    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4\": container with ID starting with 70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4 not found: ID does not exist" containerID="70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4"
Jul 09 15:09:15.786109 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.786068    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4"} err="failed to get container status \"70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4\": rpc error: code = NotFound desc = could not find container \"70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4\": container with ID starting with 70a90cb063a7cb3691e23a37743e14872a2b6c0cabadfb482d5e6b26158e4dd4 not found: ID does not exist"
Jul 09 15:09:15.798857 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.798832    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"]
Jul 09 15:09:15.801369 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.801346    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d23c1d4-6d7bc494f4-78dhr"]
Jul 09 15:09:15.868822 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.868796    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:15.868822 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.868822    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:15.868954 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:15.868833    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gbpq\" (UniqueName: \"kubernetes.io/projected/15ace234-9ddb-4fe5-ad6d-7a89587b51bb-kube-api-access-7gbpq\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:17.489236 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:17.489201    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ace234-9ddb-4fe5-ad6d-7a89587b51bb" path="/var/lib/kubelet/pods/15ace234-9ddb-4fe5-ad6d-7a89587b51bb/volumes"
Jul 09 15:09:22.621279 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.621240    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"]
Jul 09 15:09:22.621815 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.621800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ace234-9ddb-4fe5-ad6d-7a89587b51bb" containerName="maas-api"
Jul 09 15:09:22.621893 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.621819    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ace234-9ddb-4fe5-ad6d-7a89587b51bb" containerName="maas-api"
Jul 09 15:09:22.621946 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.621934    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ace234-9ddb-4fe5-ad6d-7a89587b51bb" containerName="maas-api"
Jul 09 15:09:22.626593 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.626573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.630294 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.630271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c4673fb2-serving-cert\""
Jul 09 15:09:22.640045 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.640025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"]
Jul 09 15:09:22.719188 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.719166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-openshift-service-ca\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.719300 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.719194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5d8\" (UniqueName: \"kubernetes.io/projected/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-kube-api-access-dd5d8\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.719344 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.719298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.820149 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.820124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.820272 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.820162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-openshift-service-ca\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.820272 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.820186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5d8\" (UniqueName: \"kubernetes.io/projected/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-kube-api-access-dd5d8\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.820362 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:22.820290    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-serving-cert: secret "maas-api-e2e-ait-c4673fb2-serving-cert" not found
Jul 09 15:09:22.820362 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:22.820359    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls podName:7225f9ff-2c69-405c-be9d-c21c9b9c0cf1 nodeName:}" failed. No retries permitted until 2026-07-09 15:09:23.320340599 +0000 UTC m=+1764.353869071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls") pod "maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" (UID: "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1") : secret "maas-api-e2e-ait-c4673fb2-serving-cert" not found
Jul 09 15:09:22.820770 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.820750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-openshift-service-ca\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:22.830682 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:22.830659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5d8\" (UniqueName: \"kubernetes.io/projected/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-kube-api-access-dd5d8\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:23.323712 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:23.323681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:23.326058 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:23.326037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls\") pod \"maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:23.536725 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:23.536692    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:23.676586 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:23.676526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"]
Jul 09 15:09:23.679124 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:09:23.679097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7225f9ff_2c69_405c_be9d_c21c9b9c0cf1.slice/crio-d7448fc57862490b540cae191409a0193df63eacdeebc3c63c10c76153e0028b WatchSource:0}: Error finding container d7448fc57862490b540cae191409a0193df63eacdeebc3c63c10c76153e0028b: Status 404 returned error can't find the container with id d7448fc57862490b540cae191409a0193df63eacdeebc3c63c10c76153e0028b
Jul 09 15:09:23.815705 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:23.815673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" event={"ID":"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1","Type":"ContainerStarted","Data":"d7448fc57862490b540cae191409a0193df63eacdeebc3c63c10c76153e0028b"}
Jul 09 15:09:24.822154 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:24.822055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" event={"ID":"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1","Type":"ContainerStarted","Data":"ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86"}
Jul 09 15:09:24.822571 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:24.822153    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:24.841239 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:24.841193    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" podStartSLOduration=2.116760441 podStartE2EDuration="2.841179135s" totalImagesPullingTime="724.418694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:09:24.8399145 +0000 UTC m=+1765.873442997" watchObservedRunningTime="2026-07-09 15:09:24.841179135 +0000 UTC m=+1765.874707617"
Jul 09 15:09:28.040997 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.040596    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"]
Jul 09 15:09:28.040997 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.040812    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" podUID="7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" containerName="maas-api" containerID="cri-o://ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86" gracePeriod=30
Jul 09 15:09:28.305886 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.305865    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:28.368330 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.368304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls\") pod \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") "
Jul 09 15:09:28.368467 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.368364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd5d8\" (UniqueName: \"kubernetes.io/projected/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-kube-api-access-dd5d8\") pod \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") "
Jul 09 15:09:28.368467 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.368413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-openshift-service-ca\") pod \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\" (UID: \"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1\") "
Jul 09 15:09:28.368766 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.368740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" (UID: "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:09:28.370331 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.370294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" (UID: "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:09:28.370443 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.370300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-kube-api-access-dd5d8" (OuterVolumeSpecName: "kube-api-access-dd5d8") pod "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" (UID: "7225f9ff-2c69-405c-be9d-c21c9b9c0cf1"). InnerVolumeSpecName "kube-api-access-dd5d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:09:28.468849 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.468825    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd5d8\" (UniqueName: \"kubernetes.io/projected/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-kube-api-access-dd5d8\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:28.468849 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.468846    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:28.468849 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.468855    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:28.840325 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.840283    2578 generic.go:356] "Generic (PLEG): container finished" podID="7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" containerID="ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86" exitCode=0
Jul 09 15:09:28.840527 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.840360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"
Jul 09 15:09:28.840527 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.840382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" event={"ID":"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1","Type":"ContainerDied","Data":"ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86"}
Jul 09 15:09:28.840527 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.840469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx" event={"ID":"7225f9ff-2c69-405c-be9d-c21c9b9c0cf1","Type":"ContainerDied","Data":"d7448fc57862490b540cae191409a0193df63eacdeebc3c63c10c76153e0028b"}
Jul 09 15:09:28.840527 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.840493    2578 scope.go:117] "RemoveContainer" containerID="ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86"
Jul 09 15:09:28.850138 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.849957    2578 scope.go:117] "RemoveContainer" containerID="ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86"
Jul 09 15:09:28.850336 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:28.850294    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86\": container with ID starting with ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86 not found: ID does not exist" containerID="ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86"
Jul 09 15:09:28.850437 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.850344    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86"} err="failed to get container status \"ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86\": rpc error: code = NotFound desc = could not find container \"ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86\": container with ID starting with ee70fae39c97f9ed7ff64662f4e2f35624b142bfa54e60b0fef53276a087de86 not found: ID does not exist"
Jul 09 15:09:28.867217 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.867193    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"]
Jul 09 15:09:28.871077 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:28.871058    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4673fb2-6b45f7b9b4-tvhnx"]
Jul 09 15:09:29.489551 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:29.489495    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" path="/var/lib/kubelet/pods/7225f9ff-2c69-405c-be9d-c21c9b9c0cf1/volumes"
Jul 09 15:09:36.199256 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.199222    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"]
Jul 09 15:09:36.199708 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.199669    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" containerName="maas-api"
Jul 09 15:09:36.199708 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.199695    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" containerName="maas-api"
Jul 09 15:09:36.199830 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.199791    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7225f9ff-2c69-405c-be9d-c21c9b9c0cf1" containerName="maas-api"
Jul 09 15:09:36.202740 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.202722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.205784 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.205766    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-370fdbe4-serving-cert\""
Jul 09 15:09:36.213723 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.213696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"]
Jul 09 15:09:36.331207 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.331181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxhc\" (UniqueName: \"kubernetes.io/projected/3482c804-bb25-4fb0-a80d-b1fff2911b9e-kube-api-access-5fxhc\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.331324 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.331214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.331324 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.331283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482c804-bb25-4fb0-a80d-b1fff2911b9e-openshift-service-ca\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.432244 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.432214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxhc\" (UniqueName: \"kubernetes.io/projected/3482c804-bb25-4fb0-a80d-b1fff2911b9e-kube-api-access-5fxhc\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.432393 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.432249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.432393 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.432287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482c804-bb25-4fb0-a80d-b1fff2911b9e-openshift-service-ca\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.432393 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:36.432380    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-serving-cert: secret "maas-api-e2e-derive-370fdbe4-serving-cert" not found
Jul 09 15:09:36.432511 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:36.432442    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls podName:3482c804-bb25-4fb0-a80d-b1fff2911b9e nodeName:}" failed. No retries permitted until 2026-07-09 15:09:36.932424647 +0000 UTC m=+1777.965953108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls") pod "maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" (UID: "3482c804-bb25-4fb0-a80d-b1fff2911b9e") : secret "maas-api-e2e-derive-370fdbe4-serving-cert" not found
Jul 09 15:09:36.432837 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.432822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482c804-bb25-4fb0-a80d-b1fff2911b9e-openshift-service-ca\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.454617 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.454551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxhc\" (UniqueName: \"kubernetes.io/projected/3482c804-bb25-4fb0-a80d-b1fff2911b9e-kube-api-access-5fxhc\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.936375 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.936343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:36.938826 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:36.938802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls\") pod \"maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:37.114569 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:37.114534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:37.252731 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:37.252707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"]
Jul 09 15:09:37.255059 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:09:37.255033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3482c804_bb25_4fb0_a80d_b1fff2911b9e.slice/crio-7358b14cd9197d523aa72cec7658119714e41b2b37c72ba207442f56dc5e5aea WatchSource:0}: Error finding container 7358b14cd9197d523aa72cec7658119714e41b2b37c72ba207442f56dc5e5aea: Status 404 returned error can't find the container with id 7358b14cd9197d523aa72cec7658119714e41b2b37c72ba207442f56dc5e5aea
Jul 09 15:09:37.884671 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:37.884637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" event={"ID":"3482c804-bb25-4fb0-a80d-b1fff2911b9e","Type":"ContainerStarted","Data":"7358b14cd9197d523aa72cec7658119714e41b2b37c72ba207442f56dc5e5aea"}
Jul 09 15:09:38.889935 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:38.889903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" event={"ID":"3482c804-bb25-4fb0-a80d-b1fff2911b9e","Type":"ContainerStarted","Data":"5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac"}
Jul 09 15:09:38.890308 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:38.889958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:38.906452 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:38.906408    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" podStartSLOduration=1.884388449 podStartE2EDuration="2.906392912s" totalImagesPullingTime="1.022004463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:09:38.906153463 +0000 UTC m=+1779.939681963" watchObservedRunningTime="2026-07-09 15:09:38.906392912 +0000 UTC m=+1779.939921393"
Jul 09 15:09:40.884945 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:40.884912    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"]
Jul 09 15:09:40.907803 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:40.907745    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" podUID="3482c804-bb25-4fb0-a80d-b1fff2911b9e" containerName="maas-api" containerID="cri-o://5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac" gracePeriod=30
Jul 09 15:09:41.165902 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.165877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:41.270054 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.270030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482c804-bb25-4fb0-a80d-b1fff2911b9e-openshift-service-ca\") pod \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") "
Jul 09 15:09:41.270185 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.270073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls\") pod \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") "
Jul 09 15:09:41.270185 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.270099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fxhc\" (UniqueName: \"kubernetes.io/projected/3482c804-bb25-4fb0-a80d-b1fff2911b9e-kube-api-access-5fxhc\") pod \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\" (UID: \"3482c804-bb25-4fb0-a80d-b1fff2911b9e\") "
Jul 09 15:09:41.270464 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.270437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3482c804-bb25-4fb0-a80d-b1fff2911b9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3482c804-bb25-4fb0-a80d-b1fff2911b9e" (UID: "3482c804-bb25-4fb0-a80d-b1fff2911b9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:09:41.272228 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.272201    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3482c804-bb25-4fb0-a80d-b1fff2911b9e" (UID: "3482c804-bb25-4fb0-a80d-b1fff2911b9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:09:41.272358 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.272288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482c804-bb25-4fb0-a80d-b1fff2911b9e-kube-api-access-5fxhc" (OuterVolumeSpecName: "kube-api-access-5fxhc") pod "3482c804-bb25-4fb0-a80d-b1fff2911b9e" (UID: "3482c804-bb25-4fb0-a80d-b1fff2911b9e"). InnerVolumeSpecName "kube-api-access-5fxhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:09:41.371247 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.371221    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3482c804-bb25-4fb0-a80d-b1fff2911b9e-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:41.371247 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.371243    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fxhc\" (UniqueName: \"kubernetes.io/projected/3482c804-bb25-4fb0-a80d-b1fff2911b9e-kube-api-access-5fxhc\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:41.371388 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.371253    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3482c804-bb25-4fb0-a80d-b1fff2911b9e-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:09:41.913861 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.913822    2578 generic.go:356] "Generic (PLEG): container finished" podID="3482c804-bb25-4fb0-a80d-b1fff2911b9e" containerID="5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac" exitCode=0
Jul 09 15:09:41.913861 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.913857    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"
Jul 09 15:09:41.914407 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.913876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" event={"ID":"3482c804-bb25-4fb0-a80d-b1fff2911b9e","Type":"ContainerDied","Data":"5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac"}
Jul 09 15:09:41.914407 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.913930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8" event={"ID":"3482c804-bb25-4fb0-a80d-b1fff2911b9e","Type":"ContainerDied","Data":"7358b14cd9197d523aa72cec7658119714e41b2b37c72ba207442f56dc5e5aea"}
Jul 09 15:09:41.914407 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.913960    2578 scope.go:117] "RemoveContainer" containerID="5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac"
Jul 09 15:09:41.922959 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.922798    2578 scope.go:117] "RemoveContainer" containerID="5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac"
Jul 09 15:09:41.923138 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:09:41.923112    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac\": container with ID starting with 5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac not found: ID does not exist" containerID="5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac"
Jul 09 15:09:41.923194 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.923148    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac"} err="failed to get container status \"5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac\": rpc error: code = NotFound desc = could not find container \"5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac\": container with ID starting with 5ba539f041177c8db46e9b6862acff5fb4ccff207ce11378fde0ae0065fea3ac not found: ID does not exist"
Jul 09 15:09:41.931205 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.931180    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"]
Jul 09 15:09:41.934560 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:41.934538    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370fdbe4-76c6758cf9-429h8"]
Jul 09 15:09:43.489060 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:43.489022    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3482c804-bb25-4fb0-a80d-b1fff2911b9e" path="/var/lib/kubelet/pods/3482c804-bb25-4fb0-a80d-b1fff2911b9e/volumes"
Jul 09 15:09:59.436466 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:59.436417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:09:59.440349 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:09:59.440326    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:11:25.903471 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.903391    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"]
Jul 09 15:11:25.905907 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.903899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3482c804-bb25-4fb0-a80d-b1fff2911b9e" containerName="maas-api"
Jul 09 15:11:25.905907 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.903914    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3482c804-bb25-4fb0-a80d-b1fff2911b9e" containerName="maas-api"
Jul 09 15:11:25.905907 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.904016    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3482c804-bb25-4fb0-a80d-b1fff2911b9e" containerName="maas-api"
Jul 09 15:11:25.906797 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.906778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:25.910549 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.910525    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-93979c25-serving-cert\""
Jul 09 15:11:25.918804 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.918785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"]
Jul 09 15:11:25.984913 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.984886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:25.985055 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.984944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrc6\" (UniqueName: \"kubernetes.io/projected/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-kube-api-access-wtrc6\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:25.985098 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:25.985054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-openshift-service-ca\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.085742 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.085713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.085873 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.085794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrc6\" (UniqueName: \"kubernetes.io/projected/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-kube-api-access-wtrc6\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.085873 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.085835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-openshift-service-ca\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.085873 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:11:26.085853    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-serving-cert: secret "maas-api-e2e-mt-93979c25-serving-cert" not found
Jul 09 15:11:26.086154 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:11:26.085921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls podName:ddff6be3-e2d3-451c-8d6e-0720b1d8b222 nodeName:}" failed. No retries permitted until 2026-07-09 15:11:26.585905323 +0000 UTC m=+1887.619433783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls") pod "maas-api-e2e-mt-93979c25-66d945497f-p4m2q" (UID: "ddff6be3-e2d3-451c-8d6e-0720b1d8b222") : secret "maas-api-e2e-mt-93979c25-serving-cert" not found
Jul 09 15:11:26.086558 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.086537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-openshift-service-ca\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.097092 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.097072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrc6\" (UniqueName: \"kubernetes.io/projected/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-kube-api-access-wtrc6\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.590287 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.590259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.592438 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.592414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls\") pod \"maas-api-e2e-mt-93979c25-66d945497f-p4m2q\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.819158 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.819120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:26.957258 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:26.957182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"]
Jul 09 15:11:26.960088 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:11:26.960058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddff6be3_e2d3_451c_8d6e_0720b1d8b222.slice/crio-49ca2987251840e69b64db956da6cc3eb9bc42257324aeef3023e4eff57177f3 WatchSource:0}: Error finding container 49ca2987251840e69b64db956da6cc3eb9bc42257324aeef3023e4eff57177f3: Status 404 returned error can't find the container with id 49ca2987251840e69b64db956da6cc3eb9bc42257324aeef3023e4eff57177f3
Jul 09 15:11:27.392269 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:27.392236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q" event={"ID":"ddff6be3-e2d3-451c-8d6e-0720b1d8b222","Type":"ContainerStarted","Data":"49ca2987251840e69b64db956da6cc3eb9bc42257324aeef3023e4eff57177f3"}
Jul 09 15:11:28.399195 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:28.399116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q" event={"ID":"ddff6be3-e2d3-451c-8d6e-0720b1d8b222","Type":"ContainerStarted","Data":"52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e"}
Jul 09 15:11:28.399575 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:28.399250    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:28.420057 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:28.420013    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q" podStartSLOduration=2.320434137 podStartE2EDuration="3.419999077s" totalImagesPullingTime="1.09956494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:11:28.418178674 +0000 UTC m=+1889.451707160" watchObservedRunningTime="2026-07-09 15:11:28.419999077 +0000 UTC m=+1889.453527554"
Jul 09 15:11:34.409925 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:34.409891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:37.812821 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:37.812786    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"]
Jul 09 15:11:37.813294 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:37.813085    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q" podUID="ddff6be3-e2d3-451c-8d6e-0720b1d8b222" containerName="maas-api" containerID="cri-o://52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e" gracePeriod=30
Jul 09 15:11:38.062603 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.062572    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:38.091096 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.091020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-openshift-service-ca\") pod \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") "
Jul 09 15:11:38.091247 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.091116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls\") pod \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") "
Jul 09 15:11:38.091247 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.091183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtrc6\" (UniqueName: \"kubernetes.io/projected/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-kube-api-access-wtrc6\") pod \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\" (UID: \"ddff6be3-e2d3-451c-8d6e-0720b1d8b222\") "
Jul 09 15:11:38.091444 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.091413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ddff6be3-e2d3-451c-8d6e-0720b1d8b222" (UID: "ddff6be3-e2d3-451c-8d6e-0720b1d8b222"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:11:38.093264 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.093239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ddff6be3-e2d3-451c-8d6e-0720b1d8b222" (UID: "ddff6be3-e2d3-451c-8d6e-0720b1d8b222"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:11:38.093384 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.093335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-kube-api-access-wtrc6" (OuterVolumeSpecName: "kube-api-access-wtrc6") pod "ddff6be3-e2d3-451c-8d6e-0720b1d8b222" (UID: "ddff6be3-e2d3-451c-8d6e-0720b1d8b222"). InnerVolumeSpecName "kube-api-access-wtrc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:11:38.192123 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.192089    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:11:38.192123 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.192117    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:11:38.192123 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.192130    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrc6\" (UniqueName: \"kubernetes.io/projected/ddff6be3-e2d3-451c-8d6e-0720b1d8b222-kube-api-access-wtrc6\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:11:38.451063 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.451027    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddff6be3-e2d3-451c-8d6e-0720b1d8b222" containerID="52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e" exitCode=0
Jul 09 15:11:38.451255 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.451075    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"
Jul 09 15:11:38.451255 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.451118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q" event={"ID":"ddff6be3-e2d3-451c-8d6e-0720b1d8b222","Type":"ContainerDied","Data":"52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e"}
Jul 09 15:11:38.451255 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.451166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q" event={"ID":"ddff6be3-e2d3-451c-8d6e-0720b1d8b222","Type":"ContainerDied","Data":"49ca2987251840e69b64db956da6cc3eb9bc42257324aeef3023e4eff57177f3"}
Jul 09 15:11:38.451255 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.451188    2578 scope.go:117] "RemoveContainer" containerID="52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e"
Jul 09 15:11:38.460036 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.460019    2578 scope.go:117] "RemoveContainer" containerID="52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e"
Jul 09 15:11:38.460288 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:11:38.460272    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e\": container with ID starting with 52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e not found: ID does not exist" containerID="52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e"
Jul 09 15:11:38.460332 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.460298    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e"} err="failed to get container status \"52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e\": rpc error: code = NotFound desc = could not find container \"52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e\": container with ID starting with 52a25519b97b832c25a4775b69d4b9afe775c2330fcb6d817940eb1523bd909e not found: ID does not exist"
Jul 09 15:11:38.475247 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.475222    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"]
Jul 09 15:11:38.481135 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:38.481108    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93979c25-66d945497f-p4m2q"]
Jul 09 15:11:39.488571 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:11:39.488540    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddff6be3-e2d3-451c-8d6e-0720b1d8b222" path="/var/lib/kubelet/pods/ddff6be3-e2d3-451c-8d6e-0720b1d8b222/volumes"
Jul 09 15:12:05.708201 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.708167    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"]
Jul 09 15:12:05.708783 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.708542    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddff6be3-e2d3-451c-8d6e-0720b1d8b222" containerName="maas-api"
Jul 09 15:12:05.708783 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.708557    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddff6be3-e2d3-451c-8d6e-0720b1d8b222" containerName="maas-api"
Jul 09 15:12:05.708783 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.708625    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddff6be3-e2d3-451c-8d6e-0720b1d8b222" containerName="maas-api"
Jul 09 15:12:05.711489 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.711468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.714831 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.714808    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-56796089-serving-cert\""
Jul 09 15:12:05.742453 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.742429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"]
Jul 09 15:12:05.807120 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.807098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.807235 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.807131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-openshift-service-ca\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.807235 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.807156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzwr\" (UniqueName: \"kubernetes.io/projected/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-kube-api-access-tpzwr\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.908346 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.908320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.908492 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.908354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-openshift-service-ca\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.908492 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.908397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzwr\" (UniqueName: \"kubernetes.io/projected/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-kube-api-access-tpzwr\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.908613 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:12:05.908503    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-56796089-serving-cert: secret "maas-api-e2e-mt-56796089-serving-cert" not found
Jul 09 15:12:05.908613 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:12:05.908565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls podName:93a23465-578d-49a9-9fc6-b5ca5ac3bdc9 nodeName:}" failed. No retries permitted until 2026-07-09 15:12:06.408544895 +0000 UTC m=+1927.442073371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls") pod "maas-api-e2e-mt-56796089-75595544c4-jlqnz" (UID: "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9") : secret "maas-api-e2e-mt-56796089-serving-cert" not found
Jul 09 15:12:05.909202 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.909181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-openshift-service-ca\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:05.920079 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:05.920059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzwr\" (UniqueName: \"kubernetes.io/projected/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-kube-api-access-tpzwr\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:06.412952 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:06.412917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:06.415593 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:06.415563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls\") pod \"maas-api-e2e-mt-56796089-75595544c4-jlqnz\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:06.622871 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:06.622842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:06.758260 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:06.758227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"]
Jul 09 15:12:06.761333 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:12:06.761308    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a23465_578d_49a9_9fc6_b5ca5ac3bdc9.slice/crio-acb6e1b3a50348ec2bb0ca4b129954495a757c3e2bbbf3d136e523594fe5a0d8 WatchSource:0}: Error finding container acb6e1b3a50348ec2bb0ca4b129954495a757c3e2bbbf3d136e523594fe5a0d8: Status 404 returned error can't find the container with id acb6e1b3a50348ec2bb0ca4b129954495a757c3e2bbbf3d136e523594fe5a0d8
Jul 09 15:12:07.578510 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:07.578420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" event={"ID":"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9","Type":"ContainerStarted","Data":"ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f"}
Jul 09 15:12:07.578655 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:07.578553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:07.578655 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:07.578591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" event={"ID":"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9","Type":"ContainerStarted","Data":"acb6e1b3a50348ec2bb0ca4b129954495a757c3e2bbbf3d136e523594fe5a0d8"}
Jul 09 15:12:07.596408 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:07.596361    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" podStartSLOduration=2.141623433 podStartE2EDuration="2.59634712s" totalImagesPullingTime="454.723687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:12:07.595061232 +0000 UTC m=+1928.628589715" watchObservedRunningTime="2026-07-09 15:12:07.59634712 +0000 UTC m=+1928.629875603"
Jul 09 15:12:13.587947 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:13.587917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:18.781061 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.781030    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"]
Jul 09 15:12:18.784578 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.784562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:18.789212 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.789185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-56796089\"/\"e2e-lifecycle-model-56796089-kserve-self-signed-certs\""
Jul 09 15:12:18.789369 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.789252    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-56796089\"/\"default-dockercfg-4mx68\""
Jul 09 15:12:18.789369 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.789259    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-56796089\"/\"kube-root-ca.crt\""
Jul 09 15:12:18.789487 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.789188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-56796089\"/\"openshift-service-ca.crt\""
Jul 09 15:12:18.794845 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.794819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"]
Jul 09 15:12:18.910195 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.910167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6xp\" (UniqueName: \"kubernetes.io/projected/0caa2569-70a0-4633-b292-5bf6b50cfa93-kube-api-access-kn6xp\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:18.910367 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.910228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-model-cache\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:18.910367 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.910283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-home\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:18.910367 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.910356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caa2569-70a0-4633-b292-5bf6b50cfa93-tls-certs\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:18.910534 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.910383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-dshm\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:18.910534 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:18.910486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-tmp-dir\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.010935 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.010904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-model-cache\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011113 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.010946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-home\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011113 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caa2569-70a0-4633-b292-5bf6b50cfa93-tls-certs\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011113 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-dshm\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011113 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-tmp-dir\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011113 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6xp\" (UniqueName: \"kubernetes.io/projected/0caa2569-70a0-4633-b292-5bf6b50cfa93-kube-api-access-kn6xp\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011396 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-model-cache\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011458 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-home\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.011499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.011455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-tmp-dir\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.013728 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.013705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-dshm\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.013930 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.013914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caa2569-70a0-4633-b292-5bf6b50cfa93-tls-certs\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.028435 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.028410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6xp\" (UniqueName: \"kubernetes.io/projected/0caa2569-70a0-4633-b292-5bf6b50cfa93-kube-api-access-kn6xp\") pod \"e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") " pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.093749 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.093679    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:19.229444 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.229420    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"]
Jul 09 15:12:19.231372 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:12:19.231346    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0caa2569_70a0_4633_b292_5bf6b50cfa93.slice/crio-8e240e326f72eff193192161bf804291794c402114be12feeddfb80c865118ae WatchSource:0}: Error finding container 8e240e326f72eff193192161bf804291794c402114be12feeddfb80c865118ae: Status 404 returned error can't find the container with id 8e240e326f72eff193192161bf804291794c402114be12feeddfb80c865118ae
Jul 09 15:12:19.635264 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.635232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l" event={"ID":"0caa2569-70a0-4633-b292-5bf6b50cfa93","Type":"ContainerStarted","Data":"4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b"}
Jul 09 15:12:19.635264 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.635269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l" event={"ID":"0caa2569-70a0-4633-b292-5bf6b50cfa93","Type":"ContainerStarted","Data":"8e240e326f72eff193192161bf804291794c402114be12feeddfb80c865118ae"}
Jul 09 15:12:19.657709 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:19.657668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l" podStartSLOduration=1.657652971 podStartE2EDuration="1.657652971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:12:19.655558646 +0000 UTC m=+1940.689087128" watchObservedRunningTime="2026-07-09 15:12:19.657652971 +0000 UTC m=+1940.691181456"
Jul 09 15:12:29.094619 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:29.094586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:29.095054 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:29.094635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:29.107092 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:29.107073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:29.690660 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:29.690629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:12:43.443420 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.443389    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"]
Jul 09 15:12:43.443908 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.443583    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" podUID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" containerName="maas-api" containerID="cri-o://ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f" gracePeriod=30
Jul 09 15:12:43.583732 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.583688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" podUID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.50:8443/health\": dial tcp 10.132.0.50:8443: connect: connection refused"
Jul 09 15:12:43.688610 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.688585    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:43.710708 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.710032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-openshift-service-ca\") pod \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") "
Jul 09 15:12:43.710708 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.710108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls\") pod \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") "
Jul 09 15:12:43.710708 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.710149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpzwr\" (UniqueName: \"kubernetes.io/projected/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-kube-api-access-tpzwr\") pod \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\" (UID: \"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9\") "
Jul 09 15:12:43.711019 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.710955    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" (UID: "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:12:43.714269 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.714018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" (UID: "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:12:43.714269 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.714245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-kube-api-access-tpzwr" (OuterVolumeSpecName: "kube-api-access-tpzwr") pod "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" (UID: "93a23465-578d-49a9-9fc6-b5ca5ac3bdc9"). InnerVolumeSpecName "kube-api-access-tpzwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:12:43.754092 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.754068    2578 generic.go:356] "Generic (PLEG): container finished" podID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" containerID="ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f" exitCode=0
Jul 09 15:12:43.754194 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.754102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" event={"ID":"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9","Type":"ContainerDied","Data":"ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f"}
Jul 09 15:12:43.754194 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.754113    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"
Jul 09 15:12:43.754194 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.754137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz" event={"ID":"93a23465-578d-49a9-9fc6-b5ca5ac3bdc9","Type":"ContainerDied","Data":"acb6e1b3a50348ec2bb0ca4b129954495a757c3e2bbbf3d136e523594fe5a0d8"}
Jul 09 15:12:43.754194 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.754157    2578 scope.go:117] "RemoveContainer" containerID="ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f"
Jul 09 15:12:43.763098 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.763081    2578 scope.go:117] "RemoveContainer" containerID="ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f"
Jul 09 15:12:43.763346 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:12:43.763327    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f\": container with ID starting with ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f not found: ID does not exist" containerID="ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f"
Jul 09 15:12:43.763401 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.763354    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f"} err="failed to get container status \"ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f\": rpc error: code = NotFound desc = could not find container \"ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f\": container with ID starting with ce2c785e5196921206c2cb553993ff60485fed4062cf1759f6b306b2cddb471f not found: ID does not exist"
Jul 09 15:12:43.795211 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.795193    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"]
Jul 09 15:12:43.810796 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.810772    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56796089-75595544c4-jlqnz"]
Jul 09 15:12:43.811076 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.811051    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:12:43.811076 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.811077    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpzwr\" (UniqueName: \"kubernetes.io/projected/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-kube-api-access-tpzwr\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:12:43.811218 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:43.811091    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:12:45.489109 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:45.489076    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" path="/var/lib/kubelet/pods/93a23465-578d-49a9-9fc6-b5ca5ac3bdc9/volumes"
Jul 09 15:12:53.999150 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:12:53.999079    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"]
Jul 09 15:13:09.021509 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.021459    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l" podUID="0caa2569-70a0-4633-b292-5bf6b50cfa93" containerName="main" containerID="cri-o://4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b" gracePeriod=45
Jul 09 15:13:09.261034 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.261012    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:13:09.301314 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301247    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-dshm\") pod \"0caa2569-70a0-4633-b292-5bf6b50cfa93\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") "
Jul 09 15:13:09.301314 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn6xp\" (UniqueName: \"kubernetes.io/projected/0caa2569-70a0-4633-b292-5bf6b50cfa93-kube-api-access-kn6xp\") pod \"0caa2569-70a0-4633-b292-5bf6b50cfa93\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") "
Jul 09 15:13:09.301499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-tmp-dir\") pod \"0caa2569-70a0-4633-b292-5bf6b50cfa93\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") "
Jul 09 15:13:09.301499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-model-cache\") pod \"0caa2569-70a0-4633-b292-5bf6b50cfa93\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") "
Jul 09 15:13:09.301499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-home\") pod \"0caa2569-70a0-4633-b292-5bf6b50cfa93\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") "
Jul 09 15:13:09.301499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caa2569-70a0-4633-b292-5bf6b50cfa93-tls-certs\") pod \"0caa2569-70a0-4633-b292-5bf6b50cfa93\" (UID: \"0caa2569-70a0-4633-b292-5bf6b50cfa93\") "
Jul 09 15:13:09.301705 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0caa2569-70a0-4633-b292-5bf6b50cfa93" (UID: "0caa2569-70a0-4633-b292-5bf6b50cfa93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:13:09.301818 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301775    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:13:09.301818 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-home" (OuterVolumeSpecName: "home") pod "0caa2569-70a0-4633-b292-5bf6b50cfa93" (UID: "0caa2569-70a0-4633-b292-5bf6b50cfa93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:13:09.301818 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.301807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-model-cache" (OuterVolumeSpecName: "model-cache") pod "0caa2569-70a0-4633-b292-5bf6b50cfa93" (UID: "0caa2569-70a0-4633-b292-5bf6b50cfa93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:13:09.303681 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.303644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-dshm" (OuterVolumeSpecName: "dshm") pod "0caa2569-70a0-4633-b292-5bf6b50cfa93" (UID: "0caa2569-70a0-4633-b292-5bf6b50cfa93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:13:09.303787 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.303733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0caa2569-70a0-4633-b292-5bf6b50cfa93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0caa2569-70a0-4633-b292-5bf6b50cfa93" (UID: "0caa2569-70a0-4633-b292-5bf6b50cfa93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:13:09.303902 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.303875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0caa2569-70a0-4633-b292-5bf6b50cfa93-kube-api-access-kn6xp" (OuterVolumeSpecName: "kube-api-access-kn6xp") pod "0caa2569-70a0-4633-b292-5bf6b50cfa93" (UID: "0caa2569-70a0-4633-b292-5bf6b50cfa93"). InnerVolumeSpecName "kube-api-access-kn6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:13:09.402728 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.402706    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:13:09.402728 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.402726    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:13:09.402728 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.402752    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0caa2569-70a0-4633-b292-5bf6b50cfa93-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:13:09.402952 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.402760    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0caa2569-70a0-4633-b292-5bf6b50cfa93-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:13:09.402952 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.402768    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6xp\" (UniqueName: \"kubernetes.io/projected/0caa2569-70a0-4633-b292-5bf6b50cfa93-kube-api-access-kn6xp\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:13:09.874190 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.874152    2578 generic.go:356] "Generic (PLEG): container finished" podID="0caa2569-70a0-4633-b292-5bf6b50cfa93" containerID="4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b" exitCode=0
Jul 09 15:13:09.874353 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.874194    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"
Jul 09 15:13:09.874353 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.874238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l" event={"ID":"0caa2569-70a0-4633-b292-5bf6b50cfa93","Type":"ContainerDied","Data":"4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b"}
Jul 09 15:13:09.874353 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.874273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l" event={"ID":"0caa2569-70a0-4633-b292-5bf6b50cfa93","Type":"ContainerDied","Data":"8e240e326f72eff193192161bf804291794c402114be12feeddfb80c865118ae"}
Jul 09 15:13:09.874353 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.874288    2578 scope.go:117] "RemoveContainer" containerID="4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b"
Jul 09 15:13:09.882562 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.882540    2578 scope.go:117] "RemoveContainer" containerID="4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b"
Jul 09 15:13:09.882790 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:13:09.882774    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b\": container with ID starting with 4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b not found: ID does not exist" containerID="4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b"
Jul 09 15:13:09.882848 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.882798    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b"} err="failed to get container status \"4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b\": rpc error: code = NotFound desc = could not find container \"4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b\": container with ID starting with 4951a532773f8c004659b4cf26527a954843ac68ac19a846ae0836b34f619f3b not found: ID does not exist"
Jul 09 15:13:09.892523 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.892500    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"]
Jul 09 15:13:09.896546 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:09.896524    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-56796089/e2e-lifecycle-model-56796089-kserve-b6b89dcd7-r6r9l"]
Jul 09 15:13:11.489238 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:13:11.489206    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0caa2569-70a0-4633-b292-5bf6b50cfa93" path="/var/lib/kubelet/pods/0caa2569-70a0-4633-b292-5bf6b50cfa93/volumes"
Jul 09 15:14:36.486620 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.486591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"]
Jul 09 15:14:36.487036 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.486925    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" containerName="maas-api"
Jul 09 15:14:36.487036 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.486937    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" containerName="maas-api"
Jul 09 15:14:36.487036 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.486955    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0caa2569-70a0-4633-b292-5bf6b50cfa93" containerName="main"
Jul 09 15:14:36.487036 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.486960    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0caa2569-70a0-4633-b292-5bf6b50cfa93" containerName="main"
Jul 09 15:14:36.487036 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.487035    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0caa2569-70a0-4633-b292-5bf6b50cfa93" containerName="main"
Jul 09 15:14:36.487196 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.487048    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a23465-578d-49a9-9fc6-b5ca5ac3bdc9" containerName="maas-api"
Jul 09 15:14:36.489907 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.489891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.492723 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.492704    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5ffef3-serving-cert\""
Jul 09 15:14:36.503082 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.503060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"]
Jul 09 15:14:36.566390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.566361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e57744b-4496-4653-ae10-6985fa6a2d88-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.566499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.566395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zbsc\" (UniqueName: \"kubernetes.io/projected/9e57744b-4496-4653-ae10-6985fa6a2d88-kube-api-access-2zbsc\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.566499 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.566429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e57744b-4496-4653-ae10-6985fa6a2d88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.667146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.667124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e57744b-4496-4653-ae10-6985fa6a2d88-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.667291 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.667156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zbsc\" (UniqueName: \"kubernetes.io/projected/9e57744b-4496-4653-ae10-6985fa6a2d88-kube-api-access-2zbsc\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.667291 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.667190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e57744b-4496-4653-ae10-6985fa6a2d88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.667819 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.667799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e57744b-4496-4653-ae10-6985fa6a2d88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.669423 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.669404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e57744b-4496-4653-ae10-6985fa6a2d88-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.681020 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.681001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zbsc\" (UniqueName: \"kubernetes.io/projected/9e57744b-4496-4653-ae10-6985fa6a2d88-kube-api-access-2zbsc\") pod \"maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.800584 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.800518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:36.943588 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.943548    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"]
Jul 09 15:14:36.946651 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:14:36.946610    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e57744b_4496_4653_ae10_6985fa6a2d88.slice/crio-d6dc965ec60c0c11642387cd171a720edc42c2ac55a4aab1bee3ee3d12a14221 WatchSource:0}: Error finding container d6dc965ec60c0c11642387cd171a720edc42c2ac55a4aab1bee3ee3d12a14221: Status 404 returned error can't find the container with id d6dc965ec60c0c11642387cd171a720edc42c2ac55a4aab1bee3ee3d12a14221
Jul 09 15:14:36.948220 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:36.948196    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 15:14:37.259122 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:37.259090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr" event={"ID":"9e57744b-4496-4653-ae10-6985fa6a2d88","Type":"ContainerStarted","Data":"d6dc965ec60c0c11642387cd171a720edc42c2ac55a4aab1bee3ee3d12a14221"}
Jul 09 15:14:38.265471 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:38.265435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr" event={"ID":"9e57744b-4496-4653-ae10-6985fa6a2d88","Type":"ContainerStarted","Data":"e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c"}
Jul 09 15:14:38.265851 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:38.265499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:38.285857 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:38.285813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr" podStartSLOduration=1.9145649790000001 podStartE2EDuration="2.285798834s" totalImagesPullingTime="371.233855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:14:38.284237052 +0000 UTC m=+2079.317765536" watchObservedRunningTime="2026-07-09 15:14:38.285798834 +0000 UTC m=+2079.319327315"
Jul 09 15:14:44.275447 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.275415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:14:44.531573 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.531485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"]
Jul 09 15:14:44.535023 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.534988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.538483 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.538464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e3e936-serving-cert\""
Jul 09 15:14:44.549546 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.549525    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"]
Jul 09 15:14:44.629697 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.629671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8s6\" (UniqueName: \"kubernetes.io/projected/13c916ab-6970-4448-8e5e-32daaee66c8c-kube-api-access-qr8s6\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.629827 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.629716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c916ab-6970-4448-8e5e-32daaee66c8c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.629827 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.629740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.730486 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.730458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8s6\" (UniqueName: \"kubernetes.io/projected/13c916ab-6970-4448-8e5e-32daaee66c8c-kube-api-access-qr8s6\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.730604 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.730501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c916ab-6970-4448-8e5e-32daaee66c8c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.730604 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.730517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.730681 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:14:44.730635    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-serving-cert: secret "maas-api-e2e-inf-b-e3e936-serving-cert" not found
Jul 09 15:14:44.730716 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:14:44.730699    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls podName:13c916ab-6970-4448-8e5e-32daaee66c8c nodeName:}" failed. No retries permitted until 2026-07-09 15:14:45.230678665 +0000 UTC m=+2086.264207128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls") pod "maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" (UID: "13c916ab-6970-4448-8e5e-32daaee66c8c") : secret "maas-api-e2e-inf-b-e3e936-serving-cert" not found
Jul 09 15:14:44.731208 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.731191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c916ab-6970-4448-8e5e-32daaee66c8c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:44.746202 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:44.746176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8s6\" (UniqueName: \"kubernetes.io/projected/13c916ab-6970-4448-8e5e-32daaee66c8c-kube-api-access-qr8s6\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:45.234305 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:45.234263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:45.236784 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:45.236755    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:45.445440 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:45.445404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:45.580398 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:45.580320    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"]
Jul 09 15:14:45.582739 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:14:45.582712    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c916ab_6970_4448_8e5e_32daaee66c8c.slice/crio-031080482aa12649849ac092a8eb561deeff5f475e343000184f1bdd3d7644d1 WatchSource:0}: Error finding container 031080482aa12649849ac092a8eb561deeff5f475e343000184f1bdd3d7644d1: Status 404 returned error can't find the container with id 031080482aa12649849ac092a8eb561deeff5f475e343000184f1bdd3d7644d1
Jul 09 15:14:46.304342 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:46.304264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" event={"ID":"13c916ab-6970-4448-8e5e-32daaee66c8c","Type":"ContainerStarted","Data":"4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b"}
Jul 09 15:14:46.304342 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:46.304323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" event={"ID":"13c916ab-6970-4448-8e5e-32daaee66c8c","Type":"ContainerStarted","Data":"031080482aa12649849ac092a8eb561deeff5f475e343000184f1bdd3d7644d1"}
Jul 09 15:14:46.304505 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:46.304394    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:46.323067 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:46.322993    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" podStartSLOduration=1.869321287 podStartE2EDuration="2.322958723s" totalImagesPullingTime="453.637436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:14:46.321933411 +0000 UTC m=+2087.355461905" watchObservedRunningTime="2026-07-09 15:14:46.322958723 +0000 UTC m=+2087.356487206"
Jul 09 15:14:52.315434 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:52.315405    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:14:54.463170 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.463139    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"]
Jul 09 15:14:54.466824 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.466802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.469764 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.469742    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5ffef3\"/\"openshift-service-ca.crt\""
Jul 09 15:14:54.471130 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.471115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5ffef3\"/\"kube-root-ca.crt\""
Jul 09 15:14:54.471192 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.471150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5ffef3\"/\"test-model-5ffef3-kserve-self-signed-certs\""
Jul 09 15:14:54.471192 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.471184    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5ffef3\"/\"default-dockercfg-v2gmn\""
Jul 09 15:14:54.478398 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.478375    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"]
Jul 09 15:14:54.611258 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.611226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d556dcd0-d0e1-4546-bbf0-7b042c820793-tls-certs\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.611402 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.611270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfd64\" (UniqueName: \"kubernetes.io/projected/d556dcd0-d0e1-4546-bbf0-7b042c820793-kube-api-access-wfd64\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.611402 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.611362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-home\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.611402 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.611396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-model-cache\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.611575 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.611429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-dshm\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.611575 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.611452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-tmp-dir\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712174 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d556dcd0-d0e1-4546-bbf0-7b042c820793-tls-certs\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712174 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfd64\" (UniqueName: \"kubernetes.io/projected/d556dcd0-d0e1-4546-bbf0-7b042c820793-kube-api-access-wfd64\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-home\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-model-cache\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-dshm\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-tmp-dir\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712776 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-home\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712776 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-tmp-dir\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.712967 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.712791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-model-cache\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.714664 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.714593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-dshm\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.715069 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.715048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d556dcd0-d0e1-4546-bbf0-7b042c820793-tls-certs\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.720712 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.720693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfd64\" (UniqueName: \"kubernetes.io/projected/d556dcd0-d0e1-4546-bbf0-7b042c820793-kube-api-access-wfd64\") pod \"test-model-5ffef3-kserve-5dc4dc4cff-jpsjk\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") " pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.776204 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.776164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:14:54.918797 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:54.918774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"]
Jul 09 15:14:54.921226 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:14:54.921199    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd556dcd0_d0e1_4546_bbf0_7b042c820793.slice/crio-90f7ad5a682c76cbec1eb51d20505f47dbab235fa9f95d98c254f54acfec9ea7 WatchSource:0}: Error finding container 90f7ad5a682c76cbec1eb51d20505f47dbab235fa9f95d98c254f54acfec9ea7: Status 404 returned error can't find the container with id 90f7ad5a682c76cbec1eb51d20505f47dbab235fa9f95d98c254f54acfec9ea7
Jul 09 15:14:55.349811 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:55.349719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk" event={"ID":"d556dcd0-d0e1-4546-bbf0-7b042c820793","Type":"ContainerStarted","Data":"08a3763c91961c74bcc03fc8818e1f3272047c059cef2a2e9d41d17843b4acaf"}
Jul 09 15:14:55.349811 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:55.349762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk" event={"ID":"d556dcd0-d0e1-4546-bbf0-7b042c820793","Type":"ContainerStarted","Data":"90f7ad5a682c76cbec1eb51d20505f47dbab235fa9f95d98c254f54acfec9ea7"}
Jul 09 15:14:55.371718 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:55.371673    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk" podStartSLOduration=1.371659499 podStartE2EDuration="1.371659499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:14:55.368791043 +0000 UTC m=+2096.402319526" watchObservedRunningTime="2026-07-09 15:14:55.371659499 +0000 UTC m=+2096.405188021"
Jul 09 15:14:59.464892 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:59.464865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:14:59.469111 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:14:59.469085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:15:00.133877 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.133839    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"]
Jul 09 15:15:00.137443 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.137427    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"]
Jul 09 15:15:00.137551 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.137534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:15:00.140441 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.140426    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"]
Jul 09 15:15:00.140559 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.140544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:15:00.145688 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.145271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:15:00.149361 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.149339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"]
Jul 09 15:15:00.152106 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.152079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"]
Jul 09 15:15:00.154079 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.154057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"]
Jul 09 15:15:00.159566 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.159547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvnpf\" (UniqueName: \"kubernetes.io/projected/17375fbd-4a14-428e-a63e-542f7ed90bc1-kube-api-access-lvnpf\") pod \"maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r\" (UID: \"17375fbd-4a14-428e-a63e-542f7ed90bc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:15:00.159670 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.159579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2bj\" (UniqueName: \"kubernetes.io/projected/296ab322-4f8b-46a2-8500-5cd86c808191-kube-api-access-vl2bj\") pod \"maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm\" (UID: \"296ab322-4f8b-46a2-8500-5cd86c808191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:15:00.159670 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.159605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2qn\" (UniqueName: \"kubernetes.io/projected/b9b7bfa8-2962-4dc7-b505-c547f45619b4-kube-api-access-ft2qn\") pod \"maas-api-key-cleanup-29726835-kxx2h\" (UID: \"b9b7bfa8-2962-4dc7-b505-c547f45619b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:15:00.260266 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.260230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvnpf\" (UniqueName: \"kubernetes.io/projected/17375fbd-4a14-428e-a63e-542f7ed90bc1-kube-api-access-lvnpf\") pod \"maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r\" (UID: \"17375fbd-4a14-428e-a63e-542f7ed90bc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:15:00.260266 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.260268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2bj\" (UniqueName: \"kubernetes.io/projected/296ab322-4f8b-46a2-8500-5cd86c808191-kube-api-access-vl2bj\") pod \"maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm\" (UID: \"296ab322-4f8b-46a2-8500-5cd86c808191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:15:00.260539 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.260287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2qn\" (UniqueName: \"kubernetes.io/projected/b9b7bfa8-2962-4dc7-b505-c547f45619b4-kube-api-access-ft2qn\") pod \"maas-api-key-cleanup-29726835-kxx2h\" (UID: \"b9b7bfa8-2962-4dc7-b505-c547f45619b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:15:00.270701 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.270672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2qn\" (UniqueName: \"kubernetes.io/projected/b9b7bfa8-2962-4dc7-b505-c547f45619b4-kube-api-access-ft2qn\") pod \"maas-api-key-cleanup-29726835-kxx2h\" (UID: \"b9b7bfa8-2962-4dc7-b505-c547f45619b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:15:00.270846 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.270794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2bj\" (UniqueName: \"kubernetes.io/projected/296ab322-4f8b-46a2-8500-5cd86c808191-kube-api-access-vl2bj\") pod \"maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm\" (UID: \"296ab322-4f8b-46a2-8500-5cd86c808191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:15:00.270890 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.270849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvnpf\" (UniqueName: \"kubernetes.io/projected/17375fbd-4a14-428e-a63e-542f7ed90bc1-kube-api-access-lvnpf\") pod \"maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r\" (UID: \"17375fbd-4a14-428e-a63e-542f7ed90bc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:15:00.460669 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.460642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:15:00.467638 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.467617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:15:00.473362 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.473335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:15:00.614928 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.614891    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"]
Jul 09 15:15:00.618736 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:15:00.618700    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b7bfa8_2962_4dc7_b505_c547f45619b4.slice/crio-8e33d909340cb22d138a893425cddf87960f6b6418ccfa2e9f7d8a13c8ca4e2d WatchSource:0}: Error finding container 8e33d909340cb22d138a893425cddf87960f6b6418ccfa2e9f7d8a13c8ca4e2d: Status 404 returned error can't find the container with id 8e33d909340cb22d138a893425cddf87960f6b6418ccfa2e9f7d8a13c8ca4e2d
Jul 09 15:15:00.843177 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.843153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"]
Jul 09 15:15:00.844759 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:15:00.844725    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17375fbd_4a14_428e_a63e_542f7ed90bc1.slice/crio-180ab7cdbd14844674fa692247b301d3d5e08da7f109e02266286bfc6e7361c0 WatchSource:0}: Error finding container 180ab7cdbd14844674fa692247b301d3d5e08da7f109e02266286bfc6e7361c0: Status 404 returned error can't find the container with id 180ab7cdbd14844674fa692247b301d3d5e08da7f109e02266286bfc6e7361c0
Jul 09 15:15:00.847964 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:00.847947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"]
Jul 09 15:15:00.849672 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:15:00.849650    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296ab322_4f8b_46a2_8500_5cd86c808191.slice/crio-f6100c2834bda8ab7d1ea65cc91756900d44e327ee114116bd23e54365b898a7 WatchSource:0}: Error finding container f6100c2834bda8ab7d1ea65cc91756900d44e327ee114116bd23e54365b898a7: Status 404 returned error can't find the container with id f6100c2834bda8ab7d1ea65cc91756900d44e327ee114116bd23e54365b898a7
Jul 09 15:15:01.382769 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.382682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm" event={"ID":"296ab322-4f8b-46a2-8500-5cd86c808191","Type":"ContainerStarted","Data":"a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377"}
Jul 09 15:15:01.382769 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.382740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm" event={"ID":"296ab322-4f8b-46a2-8500-5cd86c808191","Type":"ContainerStarted","Data":"f6100c2834bda8ab7d1ea65cc91756900d44e327ee114116bd23e54365b898a7"}
Jul 09 15:15:01.384136 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.384111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r" event={"ID":"17375fbd-4a14-428e-a63e-542f7ed90bc1","Type":"ContainerStarted","Data":"fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c"}
Jul 09 15:15:01.384260 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.384170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r" event={"ID":"17375fbd-4a14-428e-a63e-542f7ed90bc1","Type":"ContainerStarted","Data":"180ab7cdbd14844674fa692247b301d3d5e08da7f109e02266286bfc6e7361c0"}
Jul 09 15:15:01.385444 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.385417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h" event={"ID":"b9b7bfa8-2962-4dc7-b505-c547f45619b4","Type":"ContainerStarted","Data":"8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c"}
Jul 09 15:15:01.385551 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.385465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h" event={"ID":"b9b7bfa8-2962-4dc7-b505-c547f45619b4","Type":"ContainerStarted","Data":"8e33d909340cb22d138a893425cddf87960f6b6418ccfa2e9f7d8a13c8ca4e2d"}
Jul 09 15:15:01.399819 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.399773    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm" podStartSLOduration=1.399759389 podStartE2EDuration="1.399759389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:15:01.398777691 +0000 UTC m=+2102.432306198" watchObservedRunningTime="2026-07-09 15:15:01.399759389 +0000 UTC m=+2102.433287870"
Jul 09 15:15:01.414411 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.414369    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h" podStartSLOduration=1.414359811 podStartE2EDuration="1.414359811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:15:01.4138777 +0000 UTC m=+2102.447406182" watchObservedRunningTime="2026-07-09 15:15:01.414359811 +0000 UTC m=+2102.447888293"
Jul 09 15:15:01.430030 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:01.429989    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r" podStartSLOduration=1.429963948 podStartE2EDuration="1.429963948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:15:01.428627324 +0000 UTC m=+2102.462155807" watchObservedRunningTime="2026-07-09 15:15:01.429963948 +0000 UTC m=+2102.463492431"
Jul 09 15:15:04.777125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:04.777094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:15:04.777699 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:04.777253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:15:04.789358 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:04.789335    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:15:05.419497 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:15:05.419466    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:16:00.010725 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:00.010648    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"]
Jul 09 15:16:00.011268 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:00.010870    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm" podUID="296ab322-4f8b-46a2-8500-5cd86c808191" containerName="cleanup" containerID="cri-o://a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377" gracePeriod=30
Jul 09 15:16:00.016295 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:00.016264    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"]
Jul 09 15:16:00.016522 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:00.016498    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r" podUID="17375fbd-4a14-428e-a63e-542f7ed90bc1" containerName="cleanup" containerID="cri-o://fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c" gracePeriod=30
Jul 09 15:16:00.017895 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:00.017874    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"]
Jul 09 15:16:00.018100 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:00.018060    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h" podUID="b9b7bfa8-2962-4dc7-b505-c547f45619b4" containerName="cleanup" containerID="cri-o://8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c" gracePeriod=30
Jul 09 15:16:13.649605 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:13.649564    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"]
Jul 09 15:16:15.478748 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.478715    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"]
Jul 09 15:16:15.479238 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.478951    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr" podUID="9e57744b-4496-4653-ae10-6985fa6a2d88" containerName="maas-api" containerID="cri-o://e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c" gracePeriod=30
Jul 09 15:16:15.734652 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.734599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:16:15.734840 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.734817    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e57744b-4496-4653-ae10-6985fa6a2d88" containerID="e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c" exitCode=0
Jul 09 15:16:15.734909 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.734889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr" event={"ID":"9e57744b-4496-4653-ae10-6985fa6a2d88","Type":"ContainerDied","Data":"e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c"}
Jul 09 15:16:15.734957 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.734946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr" event={"ID":"9e57744b-4496-4653-ae10-6985fa6a2d88","Type":"ContainerDied","Data":"d6dc965ec60c0c11642387cd171a720edc42c2ac55a4aab1bee3ee3d12a14221"}
Jul 09 15:16:15.735009 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.734971    2578 scope.go:117] "RemoveContainer" containerID="e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c"
Jul 09 15:16:15.743087 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.743071    2578 scope.go:117] "RemoveContainer" containerID="e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c"
Jul 09 15:16:15.743335 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:16:15.743316    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c\": container with ID starting with e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c not found: ID does not exist" containerID="e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c"
Jul 09 15:16:15.743397 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.743343    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c"} err="failed to get container status \"e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c\": rpc error: code = NotFound desc = could not find container \"e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c\": container with ID starting with e0d714b4ae6bdf95d1e8979f22f516744b6ff0470845d98c83dc53c7fb10714c not found: ID does not exist"
Jul 09 15:16:15.748605 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.748591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zbsc\" (UniqueName: \"kubernetes.io/projected/9e57744b-4496-4653-ae10-6985fa6a2d88-kube-api-access-2zbsc\") pod \"9e57744b-4496-4653-ae10-6985fa6a2d88\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") "
Jul 09 15:16:15.748695 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.748685    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e57744b-4496-4653-ae10-6985fa6a2d88-openshift-service-ca\") pod \"9e57744b-4496-4653-ae10-6985fa6a2d88\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") "
Jul 09 15:16:15.748734 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.748718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e57744b-4496-4653-ae10-6985fa6a2d88-maas-api-tls\") pod \"9e57744b-4496-4653-ae10-6985fa6a2d88\" (UID: \"9e57744b-4496-4653-ae10-6985fa6a2d88\") "
Jul 09 15:16:15.749047 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.749027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e57744b-4496-4653-ae10-6985fa6a2d88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e57744b-4496-4653-ae10-6985fa6a2d88" (UID: "9e57744b-4496-4653-ae10-6985fa6a2d88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:16:15.750552 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.750525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e57744b-4496-4653-ae10-6985fa6a2d88-kube-api-access-2zbsc" (OuterVolumeSpecName: "kube-api-access-2zbsc") pod "9e57744b-4496-4653-ae10-6985fa6a2d88" (UID: "9e57744b-4496-4653-ae10-6985fa6a2d88"). InnerVolumeSpecName "kube-api-access-2zbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:16:15.750638 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.750576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e57744b-4496-4653-ae10-6985fa6a2d88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e57744b-4496-4653-ae10-6985fa6a2d88" (UID: "9e57744b-4496-4653-ae10-6985fa6a2d88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:16:15.849335 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.849310    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e57744b-4496-4653-ae10-6985fa6a2d88-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:15.849335 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.849335    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e57744b-4496-4653-ae10-6985fa6a2d88-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:15.849487 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:15.849346    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zbsc\" (UniqueName: \"kubernetes.io/projected/9e57744b-4496-4653-ae10-6985fa6a2d88-kube-api-access-2zbsc\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:16.740380 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:16.740351    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"
Jul 09 15:16:16.761883 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:16.761854    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"]
Jul 09 15:16:16.770654 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:16.770634    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ffef3-f84d6f7d7-jwqhr"]
Jul 09 15:16:17.488698 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:17.488663    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e57744b-4496-4653-ae10-6985fa6a2d88" path="/var/lib/kubelet/pods/9e57744b-4496-4653-ae10-6985fa6a2d88/volumes"
Jul 09 15:16:28.669851 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:28.669806    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk" podUID="d556dcd0-d0e1-4546-bbf0-7b042c820793" containerName="main" containerID="cri-o://08a3763c91961c74bcc03fc8818e1f3272047c059cef2a2e9d41d17843b4acaf" gracePeriod=45
Jul 09 15:16:28.794787 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:28.794740    2578 generic.go:356] "Generic (PLEG): container finished" podID="d556dcd0-d0e1-4546-bbf0-7b042c820793" containerID="08a3763c91961c74bcc03fc8818e1f3272047c059cef2a2e9d41d17843b4acaf" exitCode=0
Jul 09 15:16:28.794914 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:28.794806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk" event={"ID":"d556dcd0-d0e1-4546-bbf0-7b042c820793","Type":"ContainerDied","Data":"08a3763c91961c74bcc03fc8818e1f3272047c059cef2a2e9d41d17843b4acaf"}
Jul 09 15:16:28.937866 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:28.937840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:16:29.051519 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-home\") pod \"d556dcd0-d0e1-4546-bbf0-7b042c820793\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") "
Jul 09 15:16:29.051658 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d556dcd0-d0e1-4546-bbf0-7b042c820793-tls-certs\") pod \"d556dcd0-d0e1-4546-bbf0-7b042c820793\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") "
Jul 09 15:16:29.051658 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-model-cache\") pod \"d556dcd0-d0e1-4546-bbf0-7b042c820793\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") "
Jul 09 15:16:29.051658 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-dshm\") pod \"d556dcd0-d0e1-4546-bbf0-7b042c820793\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") "
Jul 09 15:16:29.051658 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051653    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-tmp-dir\") pod \"d556dcd0-d0e1-4546-bbf0-7b042c820793\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") "
Jul 09 15:16:29.051877 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfd64\" (UniqueName: \"kubernetes.io/projected/d556dcd0-d0e1-4546-bbf0-7b042c820793-kube-api-access-wfd64\") pod \"d556dcd0-d0e1-4546-bbf0-7b042c820793\" (UID: \"d556dcd0-d0e1-4546-bbf0-7b042c820793\") "
Jul 09 15:16:29.051877 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051725    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-home" (OuterVolumeSpecName: "home") pod "d556dcd0-d0e1-4546-bbf0-7b042c820793" (UID: "d556dcd0-d0e1-4546-bbf0-7b042c820793"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:16:29.052014 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051916    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:29.052014 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-model-cache" (OuterVolumeSpecName: "model-cache") pod "d556dcd0-d0e1-4546-bbf0-7b042c820793" (UID: "d556dcd0-d0e1-4546-bbf0-7b042c820793"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:16:29.052014 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.051923    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d556dcd0-d0e1-4546-bbf0-7b042c820793" (UID: "d556dcd0-d0e1-4546-bbf0-7b042c820793"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:16:29.053687 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.053658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d556dcd0-d0e1-4546-bbf0-7b042c820793-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d556dcd0-d0e1-4546-bbf0-7b042c820793" (UID: "d556dcd0-d0e1-4546-bbf0-7b042c820793"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:16:29.053687 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.053678    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-dshm" (OuterVolumeSpecName: "dshm") pod "d556dcd0-d0e1-4546-bbf0-7b042c820793" (UID: "d556dcd0-d0e1-4546-bbf0-7b042c820793"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:16:29.053831 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.053704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d556dcd0-d0e1-4546-bbf0-7b042c820793-kube-api-access-wfd64" (OuterVolumeSpecName: "kube-api-access-wfd64") pod "d556dcd0-d0e1-4546-bbf0-7b042c820793" (UID: "d556dcd0-d0e1-4546-bbf0-7b042c820793"). InnerVolumeSpecName "kube-api-access-wfd64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:16:29.152920 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.152896    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:29.152920 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.152918    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:29.153085 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.152929    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfd64\" (UniqueName: \"kubernetes.io/projected/d556dcd0-d0e1-4546-bbf0-7b042c820793-kube-api-access-wfd64\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:29.153085 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.152938    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d556dcd0-d0e1-4546-bbf0-7b042c820793-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:29.153085 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.152950    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d556dcd0-d0e1-4546-bbf0-7b042c820793-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:29.802185 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.802115    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"
Jul 09 15:16:29.802185 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.802164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk" event={"ID":"d556dcd0-d0e1-4546-bbf0-7b042c820793","Type":"ContainerDied","Data":"90f7ad5a682c76cbec1eb51d20505f47dbab235fa9f95d98c254f54acfec9ea7"}
Jul 09 15:16:29.802651 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.802208    2578 scope.go:117] "RemoveContainer" containerID="08a3763c91961c74bcc03fc8818e1f3272047c059cef2a2e9d41d17843b4acaf"
Jul 09 15:16:29.818871 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.818846    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"]
Jul 09 15:16:29.825096 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:29.825074    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5ffef3/test-model-5ffef3-kserve-5dc4dc4cff-jpsjk"]
Jul 09 15:16:30.236366 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.236338    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:16:30.238392 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.238373    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:16:30.259255 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.259232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvnpf\" (UniqueName: \"kubernetes.io/projected/17375fbd-4a14-428e-a63e-542f7ed90bc1-kube-api-access-lvnpf\") pod \"17375fbd-4a14-428e-a63e-542f7ed90bc1\" (UID: \"17375fbd-4a14-428e-a63e-542f7ed90bc1\") "
Jul 09 15:16:30.259392 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.259284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl2bj\" (UniqueName: \"kubernetes.io/projected/296ab322-4f8b-46a2-8500-5cd86c808191-kube-api-access-vl2bj\") pod \"296ab322-4f8b-46a2-8500-5cd86c808191\" (UID: \"296ab322-4f8b-46a2-8500-5cd86c808191\") "
Jul 09 15:16:30.261463 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.261438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296ab322-4f8b-46a2-8500-5cd86c808191-kube-api-access-vl2bj" (OuterVolumeSpecName: "kube-api-access-vl2bj") pod "296ab322-4f8b-46a2-8500-5cd86c808191" (UID: "296ab322-4f8b-46a2-8500-5cd86c808191"). InnerVolumeSpecName "kube-api-access-vl2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:16:30.261640 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.261616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17375fbd-4a14-428e-a63e-542f7ed90bc1-kube-api-access-lvnpf" (OuterVolumeSpecName: "kube-api-access-lvnpf") pod "17375fbd-4a14-428e-a63e-542f7ed90bc1" (UID: "17375fbd-4a14-428e-a63e-542f7ed90bc1"). InnerVolumeSpecName "kube-api-access-lvnpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:16:30.264143 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.264120    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:16:30.360610 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.360534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft2qn\" (UniqueName: \"kubernetes.io/projected/b9b7bfa8-2962-4dc7-b505-c547f45619b4-kube-api-access-ft2qn\") pod \"b9b7bfa8-2962-4dc7-b505-c547f45619b4\" (UID: \"b9b7bfa8-2962-4dc7-b505-c547f45619b4\") "
Jul 09 15:16:30.360768 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.360741    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvnpf\" (UniqueName: \"kubernetes.io/projected/17375fbd-4a14-428e-a63e-542f7ed90bc1-kube-api-access-lvnpf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:30.360768 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.360754    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2bj\" (UniqueName: \"kubernetes.io/projected/296ab322-4f8b-46a2-8500-5cd86c808191-kube-api-access-vl2bj\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:30.362469 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.362439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b7bfa8-2962-4dc7-b505-c547f45619b4-kube-api-access-ft2qn" (OuterVolumeSpecName: "kube-api-access-ft2qn") pod "b9b7bfa8-2962-4dc7-b505-c547f45619b4" (UID: "b9b7bfa8-2962-4dc7-b505-c547f45619b4"). InnerVolumeSpecName "kube-api-access-ft2qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:16:30.461829 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.461801    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft2qn\" (UniqueName: \"kubernetes.io/projected/b9b7bfa8-2962-4dc7-b505-c547f45619b4-kube-api-access-ft2qn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:30.808635 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.808594    2578 generic.go:356] "Generic (PLEG): container finished" podID="17375fbd-4a14-428e-a63e-542f7ed90bc1" containerID="fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c" exitCode=137
Jul 09 15:16:30.808635 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.808634    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"
Jul 09 15:16:30.809227 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.808677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r" event={"ID":"17375fbd-4a14-428e-a63e-542f7ed90bc1","Type":"ContainerDied","Data":"fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c"}
Jul 09 15:16:30.809227 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.808736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r" event={"ID":"17375fbd-4a14-428e-a63e-542f7ed90bc1","Type":"ContainerDied","Data":"180ab7cdbd14844674fa692247b301d3d5e08da7f109e02266286bfc6e7361c0"}
Jul 09 15:16:30.809227 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.808771    2578 scope.go:117] "RemoveContainer" containerID="fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c"
Jul 09 15:16:30.810190 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.810168    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9b7bfa8-2962-4dc7-b505-c547f45619b4" containerID="8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c" exitCode=137
Jul 09 15:16:30.810275 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.810213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"
Jul 09 15:16:30.810329 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.810285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h" event={"ID":"b9b7bfa8-2962-4dc7-b505-c547f45619b4","Type":"ContainerDied","Data":"8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c"}
Jul 09 15:16:30.810380 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.810348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h" event={"ID":"b9b7bfa8-2962-4dc7-b505-c547f45619b4","Type":"ContainerDied","Data":"8e33d909340cb22d138a893425cddf87960f6b6418ccfa2e9f7d8a13c8ca4e2d"}
Jul 09 15:16:30.811889 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.811865    2578 generic.go:356] "Generic (PLEG): container finished" podID="296ab322-4f8b-46a2-8500-5cd86c808191" containerID="a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377" exitCode=137
Jul 09 15:16:30.812038 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.811904    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"
Jul 09 15:16:30.812038 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.811924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm" event={"ID":"296ab322-4f8b-46a2-8500-5cd86c808191","Type":"ContainerDied","Data":"a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377"}
Jul 09 15:16:30.812038 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.811961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm" event={"ID":"296ab322-4f8b-46a2-8500-5cd86c808191","Type":"ContainerDied","Data":"f6100c2834bda8ab7d1ea65cc91756900d44e327ee114116bd23e54365b898a7"}
Jul 09 15:16:30.818231 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.818211    2578 scope.go:117] "RemoveContainer" containerID="fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c"
Jul 09 15:16:30.818561 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:16:30.818533    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c\": container with ID starting with fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c not found: ID does not exist" containerID="fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c"
Jul 09 15:16:30.818668 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.818570    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c"} err="failed to get container status \"fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c\": rpc error: code = NotFound desc = could not find container \"fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c\": container with ID starting with fb471d481f1d10698da8bcc148e484f23b893d3b9be4d772d95778f2fda7565c not found: ID does not exist"
Jul 09 15:16:30.818668 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.818593    2578 scope.go:117] "RemoveContainer" containerID="8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c"
Jul 09 15:16:30.827066 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.826998    2578 scope.go:117] "RemoveContainer" containerID="8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c"
Jul 09 15:16:30.827271 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:16:30.827247    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c\": container with ID starting with 8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c not found: ID does not exist" containerID="8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c"
Jul 09 15:16:30.827345 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.827277    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c"} err="failed to get container status \"8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c\": rpc error: code = NotFound desc = could not find container \"8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c\": container with ID starting with 8e1c2448e5174f133ef9e7a634de2e872f656249185c9c0e39ce4146478bf31c not found: ID does not exist"
Jul 09 15:16:30.827345 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.827297    2578 scope.go:117] "RemoveContainer" containerID="a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377"
Jul 09 15:16:30.834161 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.834146    2578 scope.go:117] "RemoveContainer" containerID="a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377"
Jul 09 15:16:30.834436 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:16:30.834418    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377\": container with ID starting with a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377 not found: ID does not exist" containerID="a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377"
Jul 09 15:16:30.834497 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.834445    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377"} err="failed to get container status \"a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377\": rpc error: code = NotFound desc = could not find container \"a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377\": container with ID starting with a87066b4e181b38cd1e698136fad823915f2b37886deb8f4bde97042812f5377 not found: ID does not exist"
Jul 09 15:16:30.837090 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.837069    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"]
Jul 09 15:16:30.846001 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.845968    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5ffef3-29726835-pm57r"]
Jul 09 15:16:30.859007 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.858969    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"]
Jul 09 15:16:30.862759 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.862742    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3e936-29726835-w79pm"]
Jul 09 15:16:30.873680 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.873661    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"]
Jul 09 15:16:30.876002 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:30.875967    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726835-kxx2h"]
Jul 09 15:16:31.489348 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:31.489313    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17375fbd-4a14-428e-a63e-542f7ed90bc1" path="/var/lib/kubelet/pods/17375fbd-4a14-428e-a63e-542f7ed90bc1/volumes"
Jul 09 15:16:31.489648 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:31.489635    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296ab322-4f8b-46a2-8500-5cd86c808191" path="/var/lib/kubelet/pods/296ab322-4f8b-46a2-8500-5cd86c808191/volumes"
Jul 09 15:16:31.489930 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:31.489918    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b7bfa8-2962-4dc7-b505-c547f45619b4" path="/var/lib/kubelet/pods/b9b7bfa8-2962-4dc7-b505-c547f45619b4/volumes"
Jul 09 15:16:31.490327 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:31.490297    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d556dcd0-d0e1-4546-bbf0-7b042c820793" path="/var/lib/kubelet/pods/d556dcd0-d0e1-4546-bbf0-7b042c820793/volumes"
Jul 09 15:16:58.936291 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:58.935688    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"]
Jul 09 15:16:58.936291 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:58.935939    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" podUID="13c916ab-6970-4448-8e5e-32daaee66c8c" containerName="maas-api" containerID="cri-o://4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b" gracePeriod=30
Jul 09 15:16:59.183242 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.183220    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:16:59.278527 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.278459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls\") pod \"13c916ab-6970-4448-8e5e-32daaee66c8c\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") "
Jul 09 15:16:59.278665 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.278535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr8s6\" (UniqueName: \"kubernetes.io/projected/13c916ab-6970-4448-8e5e-32daaee66c8c-kube-api-access-qr8s6\") pod \"13c916ab-6970-4448-8e5e-32daaee66c8c\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") "
Jul 09 15:16:59.278665 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.278642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c916ab-6970-4448-8e5e-32daaee66c8c-openshift-service-ca\") pod \"13c916ab-6970-4448-8e5e-32daaee66c8c\" (UID: \"13c916ab-6970-4448-8e5e-32daaee66c8c\") "
Jul 09 15:16:59.278973 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.278947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c916ab-6970-4448-8e5e-32daaee66c8c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13c916ab-6970-4448-8e5e-32daaee66c8c" (UID: "13c916ab-6970-4448-8e5e-32daaee66c8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:16:59.280529 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.280506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13c916ab-6970-4448-8e5e-32daaee66c8c" (UID: "13c916ab-6970-4448-8e5e-32daaee66c8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:16:59.280612 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.280510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c916ab-6970-4448-8e5e-32daaee66c8c-kube-api-access-qr8s6" (OuterVolumeSpecName: "kube-api-access-qr8s6") pod "13c916ab-6970-4448-8e5e-32daaee66c8c" (UID: "13c916ab-6970-4448-8e5e-32daaee66c8c"). InnerVolumeSpecName "kube-api-access-qr8s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:16:59.379899 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.379867    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c916ab-6970-4448-8e5e-32daaee66c8c-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:59.379899 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.379897    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8s6\" (UniqueName: \"kubernetes.io/projected/13c916ab-6970-4448-8e5e-32daaee66c8c-kube-api-access-qr8s6\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:59.379899 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.379906    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c916ab-6970-4448-8e5e-32daaee66c8c-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:16:59.950118 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.950084    2578 generic.go:356] "Generic (PLEG): container finished" podID="13c916ab-6970-4448-8e5e-32daaee66c8c" containerID="4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b" exitCode=0
Jul 09 15:16:59.950518 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.950138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"
Jul 09 15:16:59.950518 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.950168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" event={"ID":"13c916ab-6970-4448-8e5e-32daaee66c8c","Type":"ContainerDied","Data":"4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b"}
Jul 09 15:16:59.950518 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.950217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh" event={"ID":"13c916ab-6970-4448-8e5e-32daaee66c8c","Type":"ContainerDied","Data":"031080482aa12649849ac092a8eb561deeff5f475e343000184f1bdd3d7644d1"}
Jul 09 15:16:59.950518 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.950241    2578 scope.go:117] "RemoveContainer" containerID="4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b"
Jul 09 15:16:59.958436 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.958419    2578 scope.go:117] "RemoveContainer" containerID="4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b"
Jul 09 15:16:59.958726 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:16:59.958708    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b\": container with ID starting with 4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b not found: ID does not exist" containerID="4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b"
Jul 09 15:16:59.958780 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.958761    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b"} err="failed to get container status \"4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b\": rpc error: code = NotFound desc = could not find container \"4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b\": container with ID starting with 4c9d642df910c2e950931c8428e86e933328a3ab8640eea25a2b88519de8d33b not found: ID does not exist"
Jul 09 15:16:59.969125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.969103    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"]
Jul 09 15:16:59.977108 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:16:59.977088    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3e936-6bc7fd6f6-x26wh"]
Jul 09 15:17:01.488642 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:01.488608    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c916ab-6970-4448-8e5e-32daaee66c8c" path="/var/lib/kubelet/pods/13c916ab-6970-4448-8e5e-32daaee66c8c/volumes"
Jul 09 15:17:05.899583 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899551    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"]
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899911    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b7bfa8-2962-4dc7-b505-c547f45619b4" containerName="cleanup"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899922    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b7bfa8-2962-4dc7-b505-c547f45619b4" containerName="cleanup"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899935    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d556dcd0-d0e1-4546-bbf0-7b042c820793" containerName="main"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d556dcd0-d0e1-4546-bbf0-7b042c820793" containerName="main"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899949    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296ab322-4f8b-46a2-8500-5cd86c808191" containerName="cleanup"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899955    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="296ab322-4f8b-46a2-8500-5cd86c808191" containerName="cleanup"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899962    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c916ab-6970-4448-8e5e-32daaee66c8c" containerName="maas-api"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.899967    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c916ab-6970-4448-8e5e-32daaee66c8c" containerName="maas-api"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900005    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17375fbd-4a14-428e-a63e-542f7ed90bc1" containerName="cleanup"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900010    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17375fbd-4a14-428e-a63e-542f7ed90bc1" containerName="cleanup"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e57744b-4496-4653-ae10-6985fa6a2d88" containerName="maas-api"
Jul 09 15:17:05.900052 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e57744b-4496-4653-ae10-6985fa6a2d88" containerName="maas-api"
Jul 09 15:17:05.900429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900078    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e57744b-4496-4653-ae10-6985fa6a2d88" containerName="maas-api"
Jul 09 15:17:05.900429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900085    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17375fbd-4a14-428e-a63e-542f7ed90bc1" containerName="cleanup"
Jul 09 15:17:05.900429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900091    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c916ab-6970-4448-8e5e-32daaee66c8c" containerName="maas-api"
Jul 09 15:17:05.900429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900099    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d556dcd0-d0e1-4546-bbf0-7b042c820793" containerName="main"
Jul 09 15:17:05.900429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900104    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="296ab322-4f8b-46a2-8500-5cd86c808191" containerName="cleanup"
Jul 09 15:17:05.900429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.900109    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b7bfa8-2962-4dc7-b505-c547f45619b4" containerName="cleanup"
Jul 09 15:17:05.902998 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.902964    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:05.906593 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.906568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 15:17:05.906593 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.906580    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8c2065-openshift-default-dockercfg-97wms\""
Jul 09 15:17:05.923026 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:05.922997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"]
Jul 09 15:17:06.029962 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.029936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-token\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030154 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.029971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-data\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030154 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61853ad5-56f7-4920-a115-c2d49c2637bc-istiod-ca-cert\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030154 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-socket\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030154 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-certs\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030371 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98cc\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-kube-api-access-k98cc\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030371 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-envoy\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030371 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-credential-socket\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.030371 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.030333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-podinfo\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131215 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-certs\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131215 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98cc\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-kube-api-access-k98cc\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131425 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-envoy\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131425 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-credential-socket\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131425 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-podinfo\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131425 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-token\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131633 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-data\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131633 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61853ad5-56f7-4920-a115-c2d49c2637bc-istiod-ca-cert\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131633 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-socket\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131790 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-credential-socket\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.131910 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-data\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.132058 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-socket\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.132058 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.131919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-certs\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.132338 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.132313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61853ad5-56f7-4920-a115-c2d49c2637bc-istiod-ca-cert\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.133536 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.133505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-envoy\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.133834 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.133818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-podinfo\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.140540 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.140517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-token\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.141206 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.141181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98cc\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-kube-api-access-k98cc\") pod \"e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") " pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.215147 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.215126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:06.348531 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.348493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"]
Jul 09 15:17:06.351202 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:17:06.351169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61853ad5_56f7_4920_a115_c2d49c2637bc.slice/crio-141b3c436080267a3269a9c47948ac916ee87d46eb181d863814a0eca2c28990 WatchSource:0}: Error finding container 141b3c436080267a3269a9c47948ac916ee87d46eb181d863814a0eca2c28990: Status 404 returned error can't find the container with id 141b3c436080267a3269a9c47948ac916ee87d46eb181d863814a0eca2c28990
Jul 09 15:17:06.982182 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:06.982148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" event={"ID":"61853ad5-56f7-4920-a115-c2d49c2637bc","Type":"ContainerStarted","Data":"141b3c436080267a3269a9c47948ac916ee87d46eb181d863814a0eca2c28990"}
Jul 09 15:17:09.717644 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:09.717610    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:09.717936 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:09.717679    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:09.717936 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:09.717709    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:09.999152 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:09.999065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" event={"ID":"61853ad5-56f7-4920-a115-c2d49c2637bc","Type":"ContainerStarted","Data":"1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"}
Jul 09 15:17:10.023928 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:10.023846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podStartSLOduration=1.659768257 podStartE2EDuration="5.023830261s" totalImagesPullingTime="3.364062004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:17:10.021361807 +0000 UTC m=+2231.054890306" watchObservedRunningTime="2026-07-09 15:17:10.023830261 +0000 UTC m=+2231.057358745"
Jul 09 15:17:10.216002 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:10.215951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:10.217368 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:10.217342    2578 patch_prober.go:28] interesting pod/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 09 15:17:10.217503 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:10.217386    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 09 15:17:11.219829 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:11.219796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:12.009234 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.009200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:12.010443 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.010422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:12.049993 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.049950    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"]
Jul 09 15:17:12.054059 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.054033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.057347 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.057322    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8c2065-serving-cert\""
Jul 09 15:17:12.070320 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.070256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"]
Jul 09 15:17:12.180704 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.180671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54797b2d-41e6-4cae-9ada-ac79351f3b0e-maas-api-tls\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.180951 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.180841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9sb\" (UniqueName: \"kubernetes.io/projected/54797b2d-41e6-4cae-9ada-ac79351f3b0e-kube-api-access-pq9sb\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.180951 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.180866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54797b2d-41e6-4cae-9ada-ac79351f3b0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.281313 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.281231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54797b2d-41e6-4cae-9ada-ac79351f3b0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.281313 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.281285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54797b2d-41e6-4cae-9ada-ac79351f3b0e-maas-api-tls\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.281814 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.281334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9sb\" (UniqueName: \"kubernetes.io/projected/54797b2d-41e6-4cae-9ada-ac79351f3b0e-kube-api-access-pq9sb\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.282033 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.282010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54797b2d-41e6-4cae-9ada-ac79351f3b0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.283668 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.283648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54797b2d-41e6-4cae-9ada-ac79351f3b0e-maas-api-tls\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.291539 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.291518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9sb\" (UniqueName: \"kubernetes.io/projected/54797b2d-41e6-4cae-9ada-ac79351f3b0e-kube-api-access-pq9sb\") pod \"maas-api-e2e-api-a-8c2065-795675597c-c6sww\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.366119 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.366092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:12.500238 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:12.500216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"]
Jul 09 15:17:12.501888 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:17:12.501853    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54797b2d_41e6_4cae_9ada_ac79351f3b0e.slice/crio-a39e08c6b69cf3107355c847314ff4726e4fedaafa36895d8f6de4e3aea2a4ad WatchSource:0}: Error finding container a39e08c6b69cf3107355c847314ff4726e4fedaafa36895d8f6de4e3aea2a4ad: Status 404 returned error can't find the container with id a39e08c6b69cf3107355c847314ff4726e4fedaafa36895d8f6de4e3aea2a4ad
Jul 09 15:17:13.014457 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:13.014414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww" event={"ID":"54797b2d-41e6-4cae-9ada-ac79351f3b0e","Type":"ContainerStarted","Data":"a39e08c6b69cf3107355c847314ff4726e4fedaafa36895d8f6de4e3aea2a4ad"}
Jul 09 15:17:14.020650 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:14.020606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww" event={"ID":"54797b2d-41e6-4cae-9ada-ac79351f3b0e","Type":"ContainerStarted","Data":"0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58"}
Jul 09 15:17:14.041116 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:14.041069    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww" podStartSLOduration=1.3582827929999999 podStartE2EDuration="2.04105324s" totalImagesPullingTime="682.770447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:17:14.040039433 +0000 UTC m=+2235.073567915" watchObservedRunningTime="2026-07-09 15:17:14.04105324 +0000 UTC m=+2235.074581768"
Jul 09 15:17:15.025599 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:15.025560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:20.615001 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.613208    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"]
Jul 09 15:17:20.626338 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.626304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.631572 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.631553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e7df1b-serving-cert\""
Jul 09 15:17:20.663849 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.663821    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"]
Jul 09 15:17:20.755224 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.755187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnlv\" (UniqueName: \"kubernetes.io/projected/7705f49b-dafc-43e6-8687-216a1d0027f3-kube-api-access-4cnlv\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.755414 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.755250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.755414 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.755354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705f49b-dafc-43e6-8687-216a1d0027f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.856406 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.856374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.856586 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.856428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705f49b-dafc-43e6-8687-216a1d0027f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.856586 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.856464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnlv\" (UniqueName: \"kubernetes.io/projected/7705f49b-dafc-43e6-8687-216a1d0027f3-kube-api-access-4cnlv\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.856586 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:20.856521    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-serving-cert: secret "maas-api-e2e-api-b-e7df1b-serving-cert" not found
Jul 09 15:17:20.856760 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:20.856595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls podName:7705f49b-dafc-43e6-8687-216a1d0027f3 nodeName:}" failed. No retries permitted until 2026-07-09 15:17:21.356581042 +0000 UTC m=+2242.390109508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls") pod "maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" (UID: "7705f49b-dafc-43e6-8687-216a1d0027f3") : secret "maas-api-e2e-api-b-e7df1b-serving-cert" not found
Jul 09 15:17:20.857189 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.857164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705f49b-dafc-43e6-8687-216a1d0027f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:20.868165 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:20.868093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnlv\" (UniqueName: \"kubernetes.io/projected/7705f49b-dafc-43e6-8687-216a1d0027f3-kube-api-access-4cnlv\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:21.037225 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:21.037196    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:21.361905 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:21.361834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:21.364309 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:21.364289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls\") pod \"maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:21.548591 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:21.548552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:21.700354 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:21.700323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"]
Jul 09 15:17:21.702445 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:17:21.702413    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7705f49b_dafc_43e6_8687_216a1d0027f3.slice/crio-b84d1cfa89336d9b9628b4a9ba497276235dbb06f1e6b28d2f4783df34729181 WatchSource:0}: Error finding container b84d1cfa89336d9b9628b4a9ba497276235dbb06f1e6b28d2f4783df34729181: Status 404 returned error can't find the container with id b84d1cfa89336d9b9628b4a9ba497276235dbb06f1e6b28d2f4783df34729181
Jul 09 15:17:22.069472 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:22.069446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" event={"ID":"7705f49b-dafc-43e6-8687-216a1d0027f3","Type":"ContainerStarted","Data":"b84d1cfa89336d9b9628b4a9ba497276235dbb06f1e6b28d2f4783df34729181"}
Jul 09 15:17:23.075501 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:23.075465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" event={"ID":"7705f49b-dafc-43e6-8687-216a1d0027f3","Type":"ContainerStarted","Data":"ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b"}
Jul 09 15:17:23.075927 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:23.075588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:23.101962 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:23.101910    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" podStartSLOduration=2.75917946 podStartE2EDuration="3.101877299s" totalImagesPullingTime="342.697839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:17:23.100570747 +0000 UTC m=+2244.134099246" watchObservedRunningTime="2026-07-09 15:17:23.101877299 +0000 UTC m=+2244.135405834"
Jul 09 15:17:24.082165 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:24.082093    2578 generic.go:356] "Generic (PLEG): container finished" podID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerID="1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e" exitCode=0
Jul 09 15:17:24.082576 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:24.082165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" event={"ID":"61853ad5-56f7-4920-a115-c2d49c2637bc","Type":"ContainerDied","Data":"1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"}
Jul 09 15:17:24.082626 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:24.082599    2578 scope.go:117] "RemoveContainer" containerID="1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"
Jul 09 15:17:24.084466 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:24.084439    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:24.084569 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:24.084504    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:24.084569 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:24.084533    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:25.089447 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:25.089406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" event={"ID":"61853ad5-56f7-4920-a115-c2d49c2637bc","Type":"ContainerStarted","Data":"0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094"}
Jul 09 15:17:25.216588 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:25.216552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:25.217002 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:25.216937    2578 patch_prober.go:28] interesting pod/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 09 15:17:25.217205 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:25.217177    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 09 15:17:26.215908 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:26.215867    2578 patch_prober.go:28] interesting pod/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 09 15:17:26.216420 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:26.215946    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 09 15:17:27.009278 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:27.009244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:27.216084 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:27.216045    2578 patch_prober.go:28] interesting pod/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 09 15:17:27.216643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:27.216609    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 09 15:17:28.216697 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:28.216232    2578 patch_prober.go:28] interesting pod/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 09 15:17:28.216697 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:28.216298    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 09 15:17:29.088074 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:29.088047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:29.219335 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:29.219305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:29.220419 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:29.220396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:33.755459 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:33.755425    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"]
Jul 09 15:17:33.755866 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:33.755706    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww" podUID="54797b2d-41e6-4cae-9ada-ac79351f3b0e" containerName="maas-api" containerID="cri-o://0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58" gracePeriod=30
Jul 09 15:17:34.002713 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.002685    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:34.069106 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.069021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq9sb\" (UniqueName: \"kubernetes.io/projected/54797b2d-41e6-4cae-9ada-ac79351f3b0e-kube-api-access-pq9sb\") pod \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") "
Jul 09 15:17:34.069106 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.069053    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54797b2d-41e6-4cae-9ada-ac79351f3b0e-openshift-service-ca\") pod \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") "
Jul 09 15:17:34.069106 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.069076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54797b2d-41e6-4cae-9ada-ac79351f3b0e-maas-api-tls\") pod \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\" (UID: \"54797b2d-41e6-4cae-9ada-ac79351f3b0e\") "
Jul 09 15:17:34.069490 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.069459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54797b2d-41e6-4cae-9ada-ac79351f3b0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54797b2d-41e6-4cae-9ada-ac79351f3b0e" (UID: "54797b2d-41e6-4cae-9ada-ac79351f3b0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:17:34.071075 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.071046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54797b2d-41e6-4cae-9ada-ac79351f3b0e-kube-api-access-pq9sb" (OuterVolumeSpecName: "kube-api-access-pq9sb") pod "54797b2d-41e6-4cae-9ada-ac79351f3b0e" (UID: "54797b2d-41e6-4cae-9ada-ac79351f3b0e"). InnerVolumeSpecName "kube-api-access-pq9sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:17:34.071222 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.071194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54797b2d-41e6-4cae-9ada-ac79351f3b0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54797b2d-41e6-4cae-9ada-ac79351f3b0e" (UID: "54797b2d-41e6-4cae-9ada-ac79351f3b0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:17:34.134168 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.134141    2578 generic.go:356] "Generic (PLEG): container finished" podID="54797b2d-41e6-4cae-9ada-ac79351f3b0e" containerID="0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58" exitCode=0
Jul 09 15:17:34.134320 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.134183    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"
Jul 09 15:17:34.134320 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.134230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww" event={"ID":"54797b2d-41e6-4cae-9ada-ac79351f3b0e","Type":"ContainerDied","Data":"0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58"}
Jul 09 15:17:34.134320 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.134286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww" event={"ID":"54797b2d-41e6-4cae-9ada-ac79351f3b0e","Type":"ContainerDied","Data":"a39e08c6b69cf3107355c847314ff4726e4fedaafa36895d8f6de4e3aea2a4ad"}
Jul 09 15:17:34.134425 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.134328    2578 scope.go:117] "RemoveContainer" containerID="0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58"
Jul 09 15:17:34.142779 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.142761    2578 scope.go:117] "RemoveContainer" containerID="0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58"
Jul 09 15:17:34.143038 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:34.143018    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58\": container with ID starting with 0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58 not found: ID does not exist" containerID="0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58"
Jul 09 15:17:34.143092 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.143048    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58"} err="failed to get container status \"0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58\": rpc error: code = NotFound desc = could not find container \"0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58\": container with ID starting with 0329ce713040687243210c683b88dc031f7cc2b78ecb40ad6b791cb91a70ad58 not found: ID does not exist"
Jul 09 15:17:34.158830 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.158808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"]
Jul 09 15:17:34.162664 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.162642    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c2065-795675597c-c6sww"]
Jul 09 15:17:34.170221 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.170196    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq9sb\" (UniqueName: \"kubernetes.io/projected/54797b2d-41e6-4cae-9ada-ac79351f3b0e-kube-api-access-pq9sb\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:34.170221 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.170220    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54797b2d-41e6-4cae-9ada-ac79351f3b0e-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:34.170331 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:34.170230    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54797b2d-41e6-4cae-9ada-ac79351f3b0e-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:35.489607 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:35.489569    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54797b2d-41e6-4cae-9ada-ac79351f3b0e" path="/var/lib/kubelet/pods/54797b2d-41e6-4cae-9ada-ac79351f3b0e/volumes"
Jul 09 15:17:39.330065 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:39.330028    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"]
Jul 09 15:17:39.330562 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:39.330234    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" containerID="cri-o://0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094" gracePeriod=30
Jul 09 15:17:40.391626 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.391590    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"]
Jul 09 15:17:40.392093 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.391937    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" podUID="7705f49b-dafc-43e6-8687-216a1d0027f3" containerName="maas-api" containerID="cri-o://ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b" gracePeriod=30
Jul 09 15:17:40.636107 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.636084    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:40.721728 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.721698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls\") pod \"7705f49b-dafc-43e6-8687-216a1d0027f3\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") "
Jul 09 15:17:40.721874 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.721768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cnlv\" (UniqueName: \"kubernetes.io/projected/7705f49b-dafc-43e6-8687-216a1d0027f3-kube-api-access-4cnlv\") pod \"7705f49b-dafc-43e6-8687-216a1d0027f3\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") "
Jul 09 15:17:40.721874 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.721821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705f49b-dafc-43e6-8687-216a1d0027f3-openshift-service-ca\") pod \"7705f49b-dafc-43e6-8687-216a1d0027f3\" (UID: \"7705f49b-dafc-43e6-8687-216a1d0027f3\") "
Jul 09 15:17:40.722206 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.722181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7705f49b-dafc-43e6-8687-216a1d0027f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7705f49b-dafc-43e6-8687-216a1d0027f3" (UID: "7705f49b-dafc-43e6-8687-216a1d0027f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:17:40.723736 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.723708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7705f49b-dafc-43e6-8687-216a1d0027f3" (UID: "7705f49b-dafc-43e6-8687-216a1d0027f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:17:40.723827 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.723808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7705f49b-dafc-43e6-8687-216a1d0027f3-kube-api-access-4cnlv" (OuterVolumeSpecName: "kube-api-access-4cnlv") pod "7705f49b-dafc-43e6-8687-216a1d0027f3" (UID: "7705f49b-dafc-43e6-8687-216a1d0027f3"). InnerVolumeSpecName "kube-api-access-4cnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:17:40.823212 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.823183    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705f49b-dafc-43e6-8687-216a1d0027f3-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:40.823212 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.823210    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnlv\" (UniqueName: \"kubernetes.io/projected/7705f49b-dafc-43e6-8687-216a1d0027f3-kube-api-access-4cnlv\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:40.823353 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:40.823226    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705f49b-dafc-43e6-8687-216a1d0027f3-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:41.171690 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.171659    2578 generic.go:356] "Generic (PLEG): container finished" podID="7705f49b-dafc-43e6-8687-216a1d0027f3" containerID="ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b" exitCode=0
Jul 09 15:17:41.171845 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.171705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"
Jul 09 15:17:41.171845 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.171733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" event={"ID":"7705f49b-dafc-43e6-8687-216a1d0027f3","Type":"ContainerDied","Data":"ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b"}
Jul 09 15:17:41.171845 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.171773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz" event={"ID":"7705f49b-dafc-43e6-8687-216a1d0027f3","Type":"ContainerDied","Data":"b84d1cfa89336d9b9628b4a9ba497276235dbb06f1e6b28d2f4783df34729181"}
Jul 09 15:17:41.171845 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.171795    2578 scope.go:117] "RemoveContainer" containerID="ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b"
Jul 09 15:17:41.180242 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.180223    2578 scope.go:117] "RemoveContainer" containerID="ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b"
Jul 09 15:17:41.180497 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:41.180472    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b\": container with ID starting with ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b not found: ID does not exist" containerID="ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b"
Jul 09 15:17:41.180540 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.180505    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b"} err="failed to get container status \"ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b\": rpc error: code = NotFound desc = could not find container \"ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b\": container with ID starting with ceccf01b90887b433215aff849cf7b6df9791198e135afd9ebbefed79ba6855b not found: ID does not exist"
Jul 09 15:17:41.197168 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.197146    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"]
Jul 09 15:17:41.201117 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.201094    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7df1b-85f74ddcf7-zk2rz"]
Jul 09 15:17:41.489297 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:41.489217    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7705f49b-dafc-43e6-8687-216a1d0027f3" path="/var/lib/kubelet/pods/7705f49b-dafc-43e6-8687-216a1d0027f3/volumes"
Jul 09 15:17:44.222465 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.222433    2578 patch_prober.go:28] interesting pod/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 15:17:44.222868 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.222491    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 15:17:44.582438 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.582417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:44.651072 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651035    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-envoy\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651225 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61853ad5-56f7-4920-a115-c2d49c2637bc-istiod-ca-cert\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651225 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-token\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651225 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651150    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-credential-socket\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651400 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-podinfo\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651400 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-data\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651400 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651330    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k98cc\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-kube-api-access-k98cc\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651400 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-socket\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651616 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-certs\") pod \"61853ad5-56f7-4920-a115-c2d49c2637bc\" (UID: \"61853ad5-56f7-4920-a115-c2d49c2637bc\") "
Jul 09 15:17:44.651616 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651494    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:17:44.652125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:17:44.652125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651735    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.652125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.651877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:17:44.652125 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.652088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61853ad5-56f7-4920-a115-c2d49c2637bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:17:44.652495 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.652474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:17:44.653933 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.653906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-kube-api-access-k98cc" (OuterVolumeSpecName: "kube-api-access-k98cc") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "kube-api-access-k98cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:17:44.654137 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.654120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:17:44.654215 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.654153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:17:44.654277 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.654244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61853ad5-56f7-4920-a115-c2d49c2637bc" (UID: "61853ad5-56f7-4920-a115-c2d49c2637bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 15:17:44.752354 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752290    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752354 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752316    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61853ad5-56f7-4920-a115-c2d49c2637bc-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752354 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752328    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752354 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752339    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752354 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752350    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752575 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752363    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98cc\" (UniqueName: \"kubernetes.io/projected/61853ad5-56f7-4920-a115-c2d49c2637bc-kube-api-access-k98cc\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752575 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752375    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:44.752575 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:44.752388    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61853ad5-56f7-4920-a115-c2d49c2637bc-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:17:45.198322 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.198291    2578 generic.go:356] "Generic (PLEG): container finished" podID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerID="0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094" exitCode=0
Jul 09 15:17:45.198484 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.198346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" event={"ID":"61853ad5-56f7-4920-a115-c2d49c2637bc","Type":"ContainerDied","Data":"0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094"}
Jul 09 15:17:45.198484 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.198404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s" event={"ID":"61853ad5-56f7-4920-a115-c2d49c2637bc","Type":"ContainerDied","Data":"141b3c436080267a3269a9c47948ac916ee87d46eb181d863814a0eca2c28990"}
Jul 09 15:17:45.198484 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.198347    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"
Jul 09 15:17:45.198484 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.198424    2578 scope.go:117] "RemoveContainer" containerID="0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094"
Jul 09 15:17:45.207326 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.207307    2578 scope.go:117] "RemoveContainer" containerID="1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"
Jul 09 15:17:45.214930 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.214910    2578 scope.go:117] "RemoveContainer" containerID="0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094"
Jul 09 15:17:45.215201 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:45.215185    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094\": container with ID starting with 0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094 not found: ID does not exist" containerID="0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094"
Jul 09 15:17:45.215258 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.215208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094"} err="failed to get container status \"0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094\": rpc error: code = NotFound desc = could not find container \"0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094\": container with ID starting with 0ef16ec1a37952d63caa97e138ff9ab8658f8d8ad37508ae9742b3baa14b7094 not found: ID does not exist"
Jul 09 15:17:45.215258 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.215223    2578 scope.go:117] "RemoveContainer" containerID="1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"
Jul 09 15:17:45.215448 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:45.215429    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e\": container with ID starting with 1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e not found: ID does not exist" containerID="1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"
Jul 09 15:17:45.215498 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.215457    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e"} err="failed to get container status \"1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e\": rpc error: code = NotFound desc = could not find container \"1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e\": container with ID starting with 1148f454ac006517120b622140044447a5ecb5d0ca95d10af9158544075de62e not found: ID does not exist"
Jul 09 15:17:45.221278 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.221255    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"]
Jul 09 15:17:45.224885 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.224866    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8c2065-openshift-default-5df89d6875-wrd4s"]
Jul 09 15:17:45.488396 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:45.488332    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" path="/var/lib/kubelet/pods/61853ad5-56f7-4920-a115-c2d49c2637bc/volumes"
Jul 09 15:17:47.291757 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.291724    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"]
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292072    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7705f49b-dafc-43e6-8687-216a1d0027f3" containerName="maas-api"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292083    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7705f49b-dafc-43e6-8687-216a1d0027f3" containerName="maas-api"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292091    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292096    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292113    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54797b2d-41e6-4cae-9ada-ac79351f3b0e" containerName="maas-api"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292118    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54797b2d-41e6-4cae-9ada-ac79351f3b0e" containerName="maas-api"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy"
Jul 09 15:17:47.292146 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292132    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy"
Jul 09 15:17:47.292386 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292188    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy"
Jul 09 15:17:47.292386 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292197    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54797b2d-41e6-4cae-9ada-ac79351f3b0e" containerName="maas-api"
Jul 09 15:17:47.292386 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.292205    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7705f49b-dafc-43e6-8687-216a1d0027f3" containerName="maas-api"
Jul 09 15:17:47.296476 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.296459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.299191 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.299173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-cad7e2-openshift-default-dockercfg-l28qb\""
Jul 09 15:17:47.299479 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.299463    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 15:17:47.323770 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.323747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"]
Jul 09 15:17:47.375409 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sr2w\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-kube-api-access-4sr2w\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375549 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-credential-socket\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375549 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-data\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375549 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istiod-ca-cert\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375650 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-certs\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375650 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-socket\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375753 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-token\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375753 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-envoy\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.375753 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.375749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-podinfo\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477084 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-podinfo\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477222 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sr2w\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-kube-api-access-4sr2w\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477222 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-credential-socket\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477222 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-data\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477379 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istiod-ca-cert\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477379 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-certs\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477379 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-socket\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477546 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-token\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477546 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-envoy\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477649 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-credential-socket\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477649 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-data\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477782 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-socket\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.477843 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.477792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-certs\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.478226 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.478201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istiod-ca-cert\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.479570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.479550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-envoy\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.479651 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.479607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-podinfo\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.488568 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.488550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-token\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.488912 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.488885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sr2w\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-kube-api-access-4sr2w\") pod \"e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") " pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.607048 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.606963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:47.740178 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.740150    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"]
Jul 09 15:17:47.742128 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:17:47.742100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae29acc_f966_4486_a6a8_a3ab179aedbf.slice/crio-108d64428a92bb563848f049fb034adebbc3bac58e3304a97883c4ca07323216 WatchSource:0}: Error finding container 108d64428a92bb563848f049fb034adebbc3bac58e3304a97883c4ca07323216: Status 404 returned error can't find the container with id 108d64428a92bb563848f049fb034adebbc3bac58e3304a97883c4ca07323216
Jul 09 15:17:47.744219 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.744190    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:47.744285 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.744250    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:47.744285 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:47.744275    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:17:48.215362 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:48.215328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" event={"ID":"5ae29acc-f966-4486-a6a8-a3ab179aedbf","Type":"ContainerStarted","Data":"53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"}
Jul 09 15:17:48.215527 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:48.215380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" event={"ID":"5ae29acc-f966-4486-a6a8-a3ab179aedbf","Type":"ContainerStarted","Data":"108d64428a92bb563848f049fb034adebbc3bac58e3304a97883c4ca07323216"}
Jul 09 15:17:48.241792 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:48.241741    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podStartSLOduration=1.241726825 podStartE2EDuration="1.241726825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:17:48.240153075 +0000 UTC m=+2269.273681560" watchObservedRunningTime="2026-07-09 15:17:48.241726825 +0000 UTC m=+2269.275255306"
Jul 09 15:17:48.608090 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:48.608012    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:48.614126 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:48.614103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:49.221101 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:49.221066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:49.222026 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:49.222008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:17:50.341510 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.341476    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"]
Jul 09 15:17:50.341874 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.341862    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61853ad5-56f7-4920-a115-c2d49c2637bc" containerName="istio-proxy"
Jul 09 15:17:50.344831 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.344816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.348846 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.348827    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-cad7e2-serving-cert\""
Jul 09 15:17:50.367418 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.367397    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"]
Jul 09 15:17:50.400469 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.400444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16370271-e6cd-4161-a489-a2a77ac45e16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.400573 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.400472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16370271-e6cd-4161-a489-a2a77ac45e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.400573 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.400500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xr7j\" (UniqueName: \"kubernetes.io/projected/16370271-e6cd-4161-a489-a2a77ac45e16-kube-api-access-6xr7j\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.501451 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.501421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16370271-e6cd-4161-a489-a2a77ac45e16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.501580 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.501464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16370271-e6cd-4161-a489-a2a77ac45e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.501580 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.501512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xr7j\" (UniqueName: \"kubernetes.io/projected/16370271-e6cd-4161-a489-a2a77ac45e16-kube-api-access-6xr7j\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.502094 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.502064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16370271-e6cd-4161-a489-a2a77ac45e16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.503841 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.503824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16370271-e6cd-4161-a489-a2a77ac45e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.510028 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.510005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xr7j\" (UniqueName: \"kubernetes.io/projected/16370271-e6cd-4161-a489-a2a77ac45e16-kube-api-access-6xr7j\") pod \"maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.655692 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.655670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:50.795195 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:50.795064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"]
Jul 09 15:17:50.797234 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:17:50.797208    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16370271_e6cd_4161_a489_a2a77ac45e16.slice/crio-2c54ecb92c5cab96acd1a815a2cc2a971edcb56cd447c7004824fa0b15083a6f WatchSource:0}: Error finding container 2c54ecb92c5cab96acd1a815a2cc2a971edcb56cd447c7004824fa0b15083a6f: Status 404 returned error can't find the container with id 2c54ecb92c5cab96acd1a815a2cc2a971edcb56cd447c7004824fa0b15083a6f
Jul 09 15:17:51.232417 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:51.232373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp" event={"ID":"16370271-e6cd-4161-a489-a2a77ac45e16","Type":"ContainerStarted","Data":"ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e"}
Jul 09 15:17:51.232558 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:51.232435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp" event={"ID":"16370271-e6cd-4161-a489-a2a77ac45e16","Type":"ContainerStarted","Data":"2c54ecb92c5cab96acd1a815a2cc2a971edcb56cd447c7004824fa0b15083a6f"}
Jul 09 15:17:51.258084 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:51.258039    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp" podStartSLOduration=0.909665243 podStartE2EDuration="1.258023782s" totalImagesPullingTime="348.358539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:17:51.255129653 +0000 UTC m=+2272.288658136" watchObservedRunningTime="2026-07-09 15:17:51.258023782 +0000 UTC m=+2272.291552262"
Jul 09 15:17:52.237856 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:52.237829    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:58.248472 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.248440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:17:58.627692 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.627607    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"]
Jul 09 15:17:58.648742 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.647603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.649338 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.649270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"]
Jul 09 15:17:58.651347 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.651301    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-996688-serving-cert\""
Jul 09 15:17:58.770534 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.770496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.770715 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.770540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.770715 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.770560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wsn\" (UniqueName: \"kubernetes.io/projected/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-kube-api-access-k2wsn\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.871791 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.871754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.872012 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.871805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.872012 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.871831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wsn\" (UniqueName: \"kubernetes.io/projected/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-kube-api-access-k2wsn\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.872012 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:58.871900    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-serving-cert: secret "maas-api-e2e-shared-b-996688-serving-cert" not found
Jul 09 15:17:58.872012 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:17:58.872005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls podName:6000f8f8-5a3c-46a9-94cd-a53e06f96bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:17:59.371961925 +0000 UTC m=+2280.405490406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls") pod "maas-api-e2e-shared-b-996688-544cb56965-vv5c5" (UID: "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2") : secret "maas-api-e2e-shared-b-996688-serving-cert" not found
Jul 09 15:17:58.872584 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.872565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:58.880949 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:58.880895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wsn\" (UniqueName: \"kubernetes.io/projected/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-kube-api-access-k2wsn\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:59.377602 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:59.377568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:59.379909 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:59.379889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls\") pod \"maas-api-e2e-shared-b-996688-544cb56965-vv5c5\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:59.564311 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:59.564275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:17:59.751849 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:17:59.751824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"]
Jul 09 15:17:59.754701 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:17:59.754674    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6000f8f8_5a3c_46a9_94cd_a53e06f96bf2.slice/crio-f228f4ca0148706b455c3f15edd3bb28dee877967d9ae18c6a47ec3b9b7d6726 WatchSource:0}: Error finding container f228f4ca0148706b455c3f15edd3bb28dee877967d9ae18c6a47ec3b9b7d6726: Status 404 returned error can't find the container with id f228f4ca0148706b455c3f15edd3bb28dee877967d9ae18c6a47ec3b9b7d6726
Jul 09 15:18:00.276957 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:00.276914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5" event={"ID":"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2","Type":"ContainerStarted","Data":"f228f4ca0148706b455c3f15edd3bb28dee877967d9ae18c6a47ec3b9b7d6726"}
Jul 09 15:18:02.288246 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:02.288211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5" event={"ID":"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2","Type":"ContainerStarted","Data":"ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51"}
Jul 09 15:18:02.288711 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:02.288285    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:18:02.306716 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:02.306670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5" podStartSLOduration=2.408599506 podStartE2EDuration="4.306653106s" totalImagesPullingTime="1.8980536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:18:02.306359415 +0000 UTC m=+2283.339887898" watchObservedRunningTime="2026-07-09 15:18:02.306653106 +0000 UTC m=+2283.340181596"
Jul 09 15:18:04.222504 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.222469    2578 patch_prober.go:28] interesting pod/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 15:18:04.222889 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.222528    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 15:18:04.299839 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.299807    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerID="53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9" exitCode=0
Jul 09 15:18:04.300003 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.299875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" event={"ID":"5ae29acc-f966-4486-a6a8-a3ab179aedbf","Type":"ContainerDied","Data":"53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"}
Jul 09 15:18:04.300295 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.300279    2578 scope.go:117] "RemoveContainer" containerID="53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"
Jul 09 15:18:04.301264 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.301240    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:18:04.301337 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.301306    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:18:04.301384 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:04.301341    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 15:18:05.319388 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:05.319347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" event={"ID":"5ae29acc-f966-4486-a6a8-a3ab179aedbf","Type":"ContainerStarted","Data":"841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595"}
Jul 09 15:18:05.607297 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:05.607209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:18:05.607467 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:05.607418    2578 patch_prober.go:28] interesting pod/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 15:18:05.607542 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:05.607472    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 15:18:06.608205 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:06.608152    2578 patch_prober.go:28] interesting pod/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 15:18:06.608796 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:06.608741    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 15:18:07.608396 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:07.608363    2578 patch_prober.go:28] interesting pod/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 15:18:07.608995 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:07.608946    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 15:18:08.299570 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:08.299541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:18:08.608230 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:08.608088    2578 patch_prober.go:28] interesting pod/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 15:18:08.608230 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:08.608180    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 15:18:09.612328 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:09.612299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:18:09.612713 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:09.612637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:18:09.613418 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:18:09.613397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:19:59.500293 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:19:59.500269    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:19:59.504480 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:19:59.504459    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jr7vg_a20a5f44-66dd-4275-8807-af25b9282aaf/ovn-acl-logging/0.log"
Jul 09 15:21:08.756104 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:08.756068    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"]
Jul 09 15:21:08.756727 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:08.756331    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp" podUID="16370271-e6cd-4161-a489-a2a77ac45e16" containerName="maas-api" containerID="cri-o://ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e" gracePeriod=30
Jul 09 15:21:09.004891 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.004870    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:21:09.091964 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.091883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xr7j\" (UniqueName: \"kubernetes.io/projected/16370271-e6cd-4161-a489-a2a77ac45e16-kube-api-access-6xr7j\") pod \"16370271-e6cd-4161-a489-a2a77ac45e16\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") "
Jul 09 15:21:09.092134 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.091992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16370271-e6cd-4161-a489-a2a77ac45e16-maas-api-tls\") pod \"16370271-e6cd-4161-a489-a2a77ac45e16\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") "
Jul 09 15:21:09.092134 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.092030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16370271-e6cd-4161-a489-a2a77ac45e16-openshift-service-ca\") pod \"16370271-e6cd-4161-a489-a2a77ac45e16\" (UID: \"16370271-e6cd-4161-a489-a2a77ac45e16\") "
Jul 09 15:21:09.092408 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.092384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16370271-e6cd-4161-a489-a2a77ac45e16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16370271-e6cd-4161-a489-a2a77ac45e16" (UID: "16370271-e6cd-4161-a489-a2a77ac45e16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:21:09.093798 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.093764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16370271-e6cd-4161-a489-a2a77ac45e16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16370271-e6cd-4161-a489-a2a77ac45e16" (UID: "16370271-e6cd-4161-a489-a2a77ac45e16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:21:09.093902 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.093881    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16370271-e6cd-4161-a489-a2a77ac45e16-kube-api-access-6xr7j" (OuterVolumeSpecName: "kube-api-access-6xr7j") pod "16370271-e6cd-4161-a489-a2a77ac45e16" (UID: "16370271-e6cd-4161-a489-a2a77ac45e16"). InnerVolumeSpecName "kube-api-access-6xr7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:21:09.168711 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.168678    2578 generic.go:356] "Generic (PLEG): container finished" podID="16370271-e6cd-4161-a489-a2a77ac45e16" containerID="ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e" exitCode=0
Jul 09 15:21:09.168828 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.168726    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"
Jul 09 15:21:09.168828 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.168770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp" event={"ID":"16370271-e6cd-4161-a489-a2a77ac45e16","Type":"ContainerDied","Data":"ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e"}
Jul 09 15:21:09.168828 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.168825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp" event={"ID":"16370271-e6cd-4161-a489-a2a77ac45e16","Type":"ContainerDied","Data":"2c54ecb92c5cab96acd1a815a2cc2a971edcb56cd447c7004824fa0b15083a6f"}
Jul 09 15:21:09.168943 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.168854    2578 scope.go:117] "RemoveContainer" containerID="ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e"
Jul 09 15:21:09.177287 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.177269    2578 scope.go:117] "RemoveContainer" containerID="ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e"
Jul 09 15:21:09.177513 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:21:09.177496    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e\": container with ID starting with ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e not found: ID does not exist" containerID="ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e"
Jul 09 15:21:09.177558 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.177526    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e"} err="failed to get container status \"ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e\": rpc error: code = NotFound desc = could not find container \"ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e\": container with ID starting with ba7b7bfe221d967aa89b53c7f22e1af74c47bd2bbd99948c7a527e3d61c9ab7e not found: ID does not exist"
Jul 09 15:21:09.190746 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.190725    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"]
Jul 09 15:21:09.193038 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.193012    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16370271-e6cd-4161-a489-a2a77ac45e16-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:09.193038 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.193030    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16370271-e6cd-4161-a489-a2a77ac45e16-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:09.193038 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.193039    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xr7j\" (UniqueName: \"kubernetes.io/projected/16370271-e6cd-4161-a489-a2a77ac45e16-kube-api-access-6xr7j\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:09.198017 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.197997    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cad7e2-7bf8cff68-zmgzp"]
Jul 09 15:21:09.488569 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:09.488529    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16370271-e6cd-4161-a489-a2a77ac45e16" path="/var/lib/kubelet/pods/16370271-e6cd-4161-a489-a2a77ac45e16/volumes"
Jul 09 15:21:14.477154 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:14.477123    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"]
Jul 09 15:21:14.477748 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:14.477335    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy" containerID="cri-o://841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595" gracePeriod=30
Jul 09 15:21:15.689542 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:15.689502    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"]
Jul 09 15:21:15.690121 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:15.689739    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5" podUID="6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" containerName="maas-api" containerID="cri-o://ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51" gracePeriod=30
Jul 09 15:21:15.938508 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:15.938485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:21:16.047661 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.047592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-openshift-service-ca\") pod \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") "
Jul 09 15:21:16.047786 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.047665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls\") pod \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") "
Jul 09 15:21:16.047786 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.047719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2wsn\" (UniqueName: \"kubernetes.io/projected/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-kube-api-access-k2wsn\") pod \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\" (UID: \"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2\") "
Jul 09 15:21:16.047943 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.047915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" (UID: "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:21:16.049572 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.049552    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" (UID: "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 15:21:16.049782 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.049762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-kube-api-access-k2wsn" (OuterVolumeSpecName: "kube-api-access-k2wsn") pod "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" (UID: "6000f8f8-5a3c-46a9-94cd-a53e06f96bf2"). InnerVolumeSpecName "kube-api-access-k2wsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:21:16.148830 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.148803    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-maas-api-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:16.148830 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.148825    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wsn\" (UniqueName: \"kubernetes.io/projected/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-kube-api-access-k2wsn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:16.148830 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.148835    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2-openshift-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:16.203204 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.203178    2578 generic.go:356] "Generic (PLEG): container finished" podID="6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" containerID="ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51" exitCode=0
Jul 09 15:21:16.203324 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.203225    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"
Jul 09 15:21:16.203324 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.203267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5" event={"ID":"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2","Type":"ContainerDied","Data":"ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51"}
Jul 09 15:21:16.203426 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.203343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5" event={"ID":"6000f8f8-5a3c-46a9-94cd-a53e06f96bf2","Type":"ContainerDied","Data":"f228f4ca0148706b455c3f15edd3bb28dee877967d9ae18c6a47ec3b9b7d6726"}
Jul 09 15:21:16.203426 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.203365    2578 scope.go:117] "RemoveContainer" containerID="ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51"
Jul 09 15:21:16.212692 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.212675    2578 scope.go:117] "RemoveContainer" containerID="ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51"
Jul 09 15:21:16.213021 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:21:16.212993    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51\": container with ID starting with ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51 not found: ID does not exist" containerID="ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51"
Jul 09 15:21:16.213131 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.213029    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51"} err="failed to get container status \"ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51\": rpc error: code = NotFound desc = could not find container \"ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51\": container with ID starting with ab4b6edf3608fe84f479eb7097463fdcb502d4ae4bf421755eb3fab096015e51 not found: ID does not exist"
Jul 09 15:21:16.234633 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.234612    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"]
Jul 09 15:21:16.239259 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:16.239239    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-996688-544cb56965-vv5c5"]
Jul 09 15:21:17.488763 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:17.488732    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" path="/var/lib/kubelet/pods/6000f8f8-5a3c-46a9-94cd-a53e06f96bf2/volumes"
Jul 09 15:21:19.720171 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.720146    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:21:19.878973 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.878885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-token\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.878973 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.878925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-podinfo\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.878973 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.878956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-certs\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879273 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sr2w\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-kube-api-access-4sr2w\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879273 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-data\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879273 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-envoy\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879273 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istiod-ca-cert\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879273 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-credential-socket\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879273 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-socket\") pod \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\" (UID: \"5ae29acc-f966-4486-a6a8-a3ab179aedbf\") "
Jul 09 15:21:19.879565 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:21:19.879565 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879480    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.879565 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:21:19.879733 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879632    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:21:19.879806 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.879782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 15:21:19.880041 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.880017    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:21:19.881382 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.881358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 15:21:19.881382 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.881367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 15:21:19.881511 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.881374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-kube-api-access-4sr2w" (OuterVolumeSpecName: "kube-api-access-4sr2w") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "kube-api-access-4sr2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:21:19.881511 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.881418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ae29acc-f966-4486-a6a8-a3ab179aedbf" (UID: "5ae29acc-f966-4486-a6a8-a3ab179aedbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 15:21:19.980503 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980480    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980503 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980500    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980509    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980520    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980528    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sr2w\" (UniqueName: \"kubernetes.io/projected/5ae29acc-f966-4486-a6a8-a3ab179aedbf-kube-api-access-4sr2w\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980537    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980545    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:19.980643 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:19.980552    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29acc-f966-4486-a6a8-a3ab179aedbf-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Jul 09 15:21:20.224859 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.224824    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerID="841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595" exitCode=0
Jul 09 15:21:20.225072 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.224868    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"
Jul 09 15:21:20.225072 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.224910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" event={"ID":"5ae29acc-f966-4486-a6a8-a3ab179aedbf","Type":"ContainerDied","Data":"841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595"}
Jul 09 15:21:20.225072 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.224989    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k" event={"ID":"5ae29acc-f966-4486-a6a8-a3ab179aedbf","Type":"ContainerDied","Data":"108d64428a92bb563848f049fb034adebbc3bac58e3304a97883c4ca07323216"}
Jul 09 15:21:20.225072 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.225026    2578 scope.go:117] "RemoveContainer" containerID="841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595"
Jul 09 15:21:20.237450 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.237434    2578 scope.go:117] "RemoveContainer" containerID="53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"
Jul 09 15:21:20.245131 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.245113    2578 scope.go:117] "RemoveContainer" containerID="841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595"
Jul 09 15:21:20.245373 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:21:20.245355    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595\": container with ID starting with 841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595 not found: ID does not exist" containerID="841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595"
Jul 09 15:21:20.245429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.245381    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595"} err="failed to get container status \"841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595\": rpc error: code = NotFound desc = could not find container \"841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595\": container with ID starting with 841a1ccae84a8367fa6b4b38bb1487c34a55ddf1ea394bd5a5f1d118f6542595 not found: ID does not exist"
Jul 09 15:21:20.245429 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.245398    2578 scope.go:117] "RemoveContainer" containerID="53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"
Jul 09 15:21:20.245619 ip-10-0-142-236 kubenswrapper[2578]: E0709 15:21:20.245600    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9\": container with ID starting with 53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9 not found: ID does not exist" containerID="53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"
Jul 09 15:21:20.245678 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.245628    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9"} err="failed to get container status \"53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9\": rpc error: code = NotFound desc = could not find container \"53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9\": container with ID starting with 53a5ffb5a5bfb6b2f01eec7365c6b569ed54f354fe6fb84aec1f394dcddb7be9 not found: ID does not exist"
Jul 09 15:21:20.249009 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.248988    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"]
Jul 09 15:21:20.253051 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:20.253033    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-cad7e2-openshift-default-79dcbb8875-w6p2k"]
Jul 09 15:21:21.490755 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:21.490711    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" path="/var/lib/kubelet/pods/5ae29acc-f966-4486-a6a8-a3ab179aedbf/volumes"
Jul 09 15:21:27.104367 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.104336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dq9wn_84c151bf-55cf-47a5-9b0c-c550b60f8633/manager/0.log"
Jul 09 15:21:27.221216 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.221185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xqz6t_62c147fe-9e8b-4025-b734-45da4ed056a8/manager/0.log"
Jul 09 15:21:27.345134 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.345104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-764f78bc44-jg4b2_9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0/manager/0.log"
Jul 09 15:21:27.460847 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.460826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8mht6_02e1b7c0-7f72-4dd3-89c1-b8e537788d75/server/0.log"
Jul 09 15:21:27.583725 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.583687    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mks58_8adf929e-0714-47e3-b4ee-bf506720931a/manager/0.log"
Jul 09 15:21:27.857744 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.857654    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd54665dd-qjncx_ff787fdf-c6f0-4676-9d4b-a197f7ba2346/copy-manifests/0.log"
Jul 09 15:21:27.865631 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:27.865607    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd54665dd-qjncx_ff787fdf-c6f0-4676-9d4b-a197f7ba2346/manager/0.log"
Jul 09 15:21:29.483365 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:29.483324    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2v7h2_1a2a78fd-09d9-4f62-820b-778acebd3656/manager/0.log"
Jul 09 15:21:29.607666 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:29.607622    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d4hh5_da51c1cb-eda5-4134-919c-0fec24b68ade/manager/0.log"
Jul 09 15:21:29.850300 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:29.850218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-b4lld_b3513bed-14aa-4b7b-bb10-c193a9437c8b/registry-server/0.log"
Jul 09 15:21:29.978574 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:29.978545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6g2rv_d9806c78-3080-4ec4-8257-98adb745853a/manager/0.log"
Jul 09 15:21:30.228510 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:30.228440    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-29bvx_a0575a99-9a05-467a-90cd-9f3b43b8f73f/manager/0.log"
Jul 09 15:21:30.833332 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:30.833305    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c698c67c6-9qkln_ae374918-32df-4b1d-859d-0f74b3ca398b/kube-auth-proxy/0.log"
Jul 09 15:21:31.345434 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:31.345382    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dlzgw_b7aea3be-e1be-4f86-8a40-61474409c4ab/payload-processing/0.log"
Jul 09 15:21:32.049876 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:32.049828    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tj6d4_e47919ce-b31c-4b77-bcf6-dce037515912/main/0.log"
Jul 09 15:21:32.167885 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:32.167847    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqnbjt_269b6d80-2780-40fe-9abf-85bfa43b5072/main/0.log"
Jul 09 15:21:36.485687 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.485654    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s264d/must-gather-dmp4c"]
Jul 09 15:21:36.486065 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486052    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16370271-e6cd-4161-a489-a2a77ac45e16" containerName="maas-api"
Jul 09 15:21:36.486065 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486064    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16370271-e6cd-4161-a489-a2a77ac45e16" containerName="maas-api"
Jul 09 15:21:36.486149 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486081    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy"
Jul 09 15:21:36.486149 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486086    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy"
Jul 09 15:21:36.486149 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486102    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" containerName="maas-api"
Jul 09 15:21:36.486149 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486107    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" containerName="maas-api"
Jul 09 15:21:36.486266 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486176    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6000f8f8-5a3c-46a9-94cd-a53e06f96bf2" containerName="maas-api"
Jul 09 15:21:36.486266 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486185    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy"
Jul 09 15:21:36.486266 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486198    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16370271-e6cd-4161-a489-a2a77ac45e16" containerName="maas-api"
Jul 09 15:21:36.486364 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486267    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy"
Jul 09 15:21:36.486364 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486275    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy"
Jul 09 15:21:36.486364 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.486333    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae29acc-f966-4486-a6a8-a3ab179aedbf" containerName="istio-proxy"
Jul 09 15:21:36.489215 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.489197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.493491 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.493470    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s264d\"/\"default-dockercfg-pspkd\""
Jul 09 15:21:36.493840 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.493821    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s264d\"/\"kube-root-ca.crt\""
Jul 09 15:21:36.493897 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.493821    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s264d\"/\"openshift-service-ca.crt\""
Jul 09 15:21:36.499885 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.499865    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s264d/must-gather-dmp4c"]
Jul 09 15:21:36.611599 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.611569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bfa99f3-540b-4100-b616-48e6dbc011fe-must-gather-output\") pod \"must-gather-dmp4c\" (UID: \"0bfa99f3-540b-4100-b616-48e6dbc011fe\") " pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.611750 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.611620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7wb\" (UniqueName: \"kubernetes.io/projected/0bfa99f3-540b-4100-b616-48e6dbc011fe-kube-api-access-dm7wb\") pod \"must-gather-dmp4c\" (UID: \"0bfa99f3-540b-4100-b616-48e6dbc011fe\") " pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.712855 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.712828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bfa99f3-540b-4100-b616-48e6dbc011fe-must-gather-output\") pod \"must-gather-dmp4c\" (UID: \"0bfa99f3-540b-4100-b616-48e6dbc011fe\") " pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.712958 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.712872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7wb\" (UniqueName: \"kubernetes.io/projected/0bfa99f3-540b-4100-b616-48e6dbc011fe-kube-api-access-dm7wb\") pod \"must-gather-dmp4c\" (UID: \"0bfa99f3-540b-4100-b616-48e6dbc011fe\") " pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.713226 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.713209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bfa99f3-540b-4100-b616-48e6dbc011fe-must-gather-output\") pod \"must-gather-dmp4c\" (UID: \"0bfa99f3-540b-4100-b616-48e6dbc011fe\") " pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.721336 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.721317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7wb\" (UniqueName: \"kubernetes.io/projected/0bfa99f3-540b-4100-b616-48e6dbc011fe-kube-api-access-dm7wb\") pod \"must-gather-dmp4c\" (UID: \"0bfa99f3-540b-4100-b616-48e6dbc011fe\") " pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.798217 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.798156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s264d/must-gather-dmp4c"
Jul 09 15:21:36.935675 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.935648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s264d/must-gather-dmp4c"]
Jul 09 15:21:36.936664 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:21:36.936639    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bfa99f3_540b_4100_b616_48e6dbc011fe.slice/crio-e263b18bd1f11ce4e7faadc17e3cf8fe88dcd5f387c3694c4be2110eb1ec11cc WatchSource:0}: Error finding container e263b18bd1f11ce4e7faadc17e3cf8fe88dcd5f387c3694c4be2110eb1ec11cc: Status 404 returned error can't find the container with id e263b18bd1f11ce4e7faadc17e3cf8fe88dcd5f387c3694c4be2110eb1ec11cc
Jul 09 15:21:36.938615 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:36.938600    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 15:21:37.304904 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:37.304872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s264d/must-gather-dmp4c" event={"ID":"0bfa99f3-540b-4100-b616-48e6dbc011fe","Type":"ContainerStarted","Data":"e263b18bd1f11ce4e7faadc17e3cf8fe88dcd5f387c3694c4be2110eb1ec11cc"}
Jul 09 15:21:38.315642 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:38.314845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s264d/must-gather-dmp4c" event={"ID":"0bfa99f3-540b-4100-b616-48e6dbc011fe","Type":"ContainerStarted","Data":"534fd66e40991cfc554394d3e9f37c0ae5c42ca5d1831982b755dc806fbe35bc"}
Jul 09 15:21:38.315642 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:38.314901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s264d/must-gather-dmp4c" event={"ID":"0bfa99f3-540b-4100-b616-48e6dbc011fe","Type":"ContainerStarted","Data":"149ab5f80824050c1693bca412fcff9e57f378deb77568508ddcedf9c26a4515"}
Jul 09 15:21:38.334877 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:38.334822    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s264d/must-gather-dmp4c" podStartSLOduration=1.464843779 podStartE2EDuration="2.33480019s" totalImagesPullingTime="869.956411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:21:38.332955878 +0000 UTC m=+2499.366484386" watchObservedRunningTime="2026-07-09 15:21:38.33480019 +0000 UTC m=+2499.368328674"
Jul 09 15:21:39.441529 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:39.441499    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w8jb7_c65bcbab-1c89-4bc0-970f-4f17177a5ed1/global-pull-secret-syncer/0.log"
Jul 09 15:21:39.522024 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:39.521941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c5htq_20081f0a-4efc-4d49-941b-e5f0713855d3/konnectivity-agent/0.log"
Jul 09 15:21:39.631541 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:39.631513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-236.ec2.internal_6d92aa44007fbeeba58b77b0f3aaedc2/haproxy/0.log"
Jul 09 15:21:44.403628 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:44.403503    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2v7h2_1a2a78fd-09d9-4f62-820b-778acebd3656/manager/0.log"
Jul 09 15:21:44.461609 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:44.461571    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d4hh5_da51c1cb-eda5-4134-919c-0fec24b68ade/manager/0.log"
Jul 09 15:21:44.570264 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:44.570232    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-b4lld_b3513bed-14aa-4b7b-bb10-c193a9437c8b/registry-server/0.log"
Jul 09 15:21:44.736910 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:44.736821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6g2rv_d9806c78-3080-4ec4-8257-98adb745853a/manager/0.log"
Jul 09 15:21:44.876949 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:44.876904    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-29bvx_a0575a99-9a05-467a-90cd-9f3b43b8f73f/manager/0.log"
Jul 09 15:21:47.235567 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:47.235537    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m449j_87831920-1c8e-4f13-8235-05f069e2178a/node-exporter/0.log"
Jul 09 15:21:47.257912 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:47.257886    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m449j_87831920-1c8e-4f13-8235-05f069e2178a/kube-rbac-proxy/0.log"
Jul 09 15:21:47.279777 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:47.279755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m449j_87831920-1c8e-4f13-8235-05f069e2178a/init-textfile/0.log"
Jul 09 15:21:48.912328 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:48.912297    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-gkb6h_a8872040-1139-4109-94fd-a878b137fb4d/networking-console-plugin/0.log"
Jul 09 15:21:49.528710 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.528671    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"]
Jul 09 15:21:49.536876 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.536845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.543716 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.543681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"]
Jul 09 15:21:49.632194 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.632138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-sys\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.632388 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.632218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-lib-modules\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.632388 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.632261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-podres\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.632388 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.632316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-proc\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.632388 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.632343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5smj\" (UniqueName: \"kubernetes.io/projected/0799fd63-5aec-485e-bb9e-4af195f1ed80-kube-api-access-w5smj\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733554 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-sys\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733729 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-lib-modules\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733729 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-podres\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733729 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-proc\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733729 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-sys\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733729 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5smj\" (UniqueName: \"kubernetes.io/projected/0799fd63-5aec-485e-bb9e-4af195f1ed80-kube-api-access-w5smj\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733958 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-proc\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733958 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-podres\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.733958 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.733786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0799fd63-5aec-485e-bb9e-4af195f1ed80-lib-modules\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.742615 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.742590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5smj\" (UniqueName: \"kubernetes.io/projected/0799fd63-5aec-485e-bb9e-4af195f1ed80-kube-api-access-w5smj\") pod \"perf-node-gather-daemonset-l2rr4\" (UID: \"0799fd63-5aec-485e-bb9e-4af195f1ed80\") " pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:49.857048 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:49.856958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:50.035216 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:50.035193    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"]
Jul 09 15:21:50.038001 ip-10-0-142-236 kubenswrapper[2578]: W0709 15:21:50.037944    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0799fd63_5aec_485e_bb9e_4af195f1ed80.slice/crio-3a566b0daecd2573bbc93533a19c3dec5a6c5e95f0b8267339e3bb57f54a5cae WatchSource:0}: Error finding container 3a566b0daecd2573bbc93533a19c3dec5a6c5e95f0b8267339e3bb57f54a5cae: Status 404 returned error can't find the container with id 3a566b0daecd2573bbc93533a19c3dec5a6c5e95f0b8267339e3bb57f54a5cae
Jul 09 15:21:50.395589 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:50.395490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4" event={"ID":"0799fd63-5aec-485e-bb9e-4af195f1ed80","Type":"ContainerStarted","Data":"fb0a7fa203ce8ea96d0c1308e52fa34ee3fc0951489aeb730e20232d8f8db52e"}
Jul 09 15:21:50.395589 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:50.395550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4" event={"ID":"0799fd63-5aec-485e-bb9e-4af195f1ed80","Type":"ContainerStarted","Data":"3a566b0daecd2573bbc93533a19c3dec5a6c5e95f0b8267339e3bb57f54a5cae"}
Jul 09 15:21:50.395589 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:50.395580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:50.414561 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:50.414499    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4" podStartSLOduration=1.414478241 podStartE2EDuration="1.414478241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:21:50.412687472 +0000 UTC m=+2511.446215954" watchObservedRunningTime="2026-07-09 15:21:50.414478241 +0000 UTC m=+2511.448006723"
Jul 09 15:21:51.640664 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:51.640591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9z945_2a823838-6515-4fb7-a952-c6626369db38/dns/0.log"
Jul 09 15:21:51.665627 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:51.665599    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9z945_2a823838-6515-4fb7-a952-c6626369db38/kube-rbac-proxy/0.log"
Jul 09 15:21:51.788055 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:51.788026    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vvm8k_4f93dcb3-a544-417e-839f-fb1fc4c85494/dns-node-resolver/0.log"
Jul 09 15:21:52.361749 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:52.361717    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-psqp4_886304ba-ee5f-4af4-9435-3a256bc1ede6/node-ca/0.log"
Jul 09 15:21:53.381440 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:53.381413    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c698c67c6-9qkln_ae374918-32df-4b1d-859d-0f74b3ca398b/kube-auth-proxy/0.log"
Jul 09 15:21:53.560048 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:53.560017    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dlzgw_b7aea3be-e1be-4f86-8a40-61474409c4ab/payload-processing/0.log"
Jul 09 15:21:54.374826 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:54.374801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-spwkt_92162f15-4468-4c81-ad96-0d5fa0967389/serve-healthcheck-canary/0.log"
Jul 09 15:21:55.054708 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:55.054681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hclkh_b577cfd6-be9c-449f-8488-87407592f9b1/kube-rbac-proxy/0.log"
Jul 09 15:21:55.077309 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:55.077284    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hclkh_b577cfd6-be9c-449f-8488-87407592f9b1/exporter/0.log"
Jul 09 15:21:55.100619 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:55.100595    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hclkh_b577cfd6-be9c-449f-8488-87407592f9b1/extractor/0.log"
Jul 09 15:21:56.413667 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:56.413635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s264d/perf-node-gather-daemonset-l2rr4"
Jul 09 15:21:57.366701 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.366675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dq9wn_84c151bf-55cf-47a5-9b0c-c550b60f8633/manager/0.log"
Jul 09 15:21:57.511027 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.510971    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xqz6t_62c147fe-9e8b-4025-b734-45da4ed056a8/manager/0.log"
Jul 09 15:21:57.622970 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.622807    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-764f78bc44-jg4b2_9ae0ebb4-4b89-479c-8f62-d0677d1a9ff0/manager/0.log"
Jul 09 15:21:57.646005 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.645965    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8mht6_02e1b7c0-7f72-4dd3-89c1-b8e537788d75/server/0.log"
Jul 09 15:21:57.726106 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.726070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mks58_8adf929e-0714-47e3-b4ee-bf506720931a/manager/0.log"
Jul 09 15:21:57.800691 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.800664    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd54665dd-qjncx_ff787fdf-c6f0-4676-9d4b-a197f7ba2346/manager/0.log"
Jul 09 15:21:57.824383 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:57.824354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd54665dd-qjncx_ff787fdf-c6f0-4676-9d4b-a197f7ba2346/copy-manifests/0.log"
Jul 09 15:21:59.244698 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:59.244656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c698c67c6-9qkln_ae374918-32df-4b1d-859d-0f74b3ca398b/kube-auth-proxy/0.log"
Jul 09 15:21:59.392547 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:21:59.392475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dlzgw_b7aea3be-e1be-4f86-8a40-61474409c4ab/payload-processing/0.log"
Jul 09 15:22:00.714972 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:00.714940    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c698c67c6-9qkln_ae374918-32df-4b1d-859d-0f74b3ca398b/kube-auth-proxy/0.log"
Jul 09 15:22:00.866866 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:00.866755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dlzgw_b7aea3be-e1be-4f86-8a40-61474409c4ab/payload-processing/0.log"
Jul 09 15:22:05.764116 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.764086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/kube-multus-additional-cni-plugins/0.log"
Jul 09 15:22:05.784456 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.784435    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/egress-router-binary-copy/0.log"
Jul 09 15:22:05.805383 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.805360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/cni-plugins/0.log"
Jul 09 15:22:05.825181 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.825160    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/bond-cni-plugin/0.log"
Jul 09 15:22:05.846050 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.846024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/routeoverride-cni/0.log"
Jul 09 15:22:05.867251 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.867231    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/whereabouts-cni-bincopy/0.log"
Jul 09 15:22:05.888313 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.888291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n7r6j_9df2893b-e304-464d-a710-8d7094a8e4d2/whereabouts-cni/0.log"
Jul 09 15:22:05.979850 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:05.979829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v22jf_2fe5c7d3-ffef-4c1c-a6f1-bda64e2895a5/kube-multus/0.log"
Jul 09 15:22:06.047390 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:06.047314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7vp7b_f26376fc-b839-4b7e-9fa7-160aa9b0faa7/network-metrics-daemon/0.log"
Jul 09 15:22:06.066179 ip-10-0-142-236 kubenswrapper[2578]: I0709 15:22:06.066157    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7vp7b_f26376fc-b839-4b7e-9fa7-160aa9b0faa7/kube-rbac-proxy/0.log"