Jul 23 17:59:21.111270 ip-10-0-130-3 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 17:59:21.426585 ip-10-0-130-3 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 17:59:21.426585 ip-10-0-130-3 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 17:59:21.426585 ip-10-0-130-3 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 17:59:21.426585 ip-10-0-130-3 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 17:59:21.426585 ip-10-0-130-3 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 17:59:21.427361 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.427269    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432920    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432952    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432956    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432959    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432962    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432964    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:59:21.432957 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432967    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432970    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432972    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432975    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432977    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432979    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432982    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432984    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432987    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432989    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432992    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432994    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432996    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.432999    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433002    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433004    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433007    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433010    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433012    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433014    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:59:21.433194 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433017    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433019    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433021    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433024    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433026    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433029    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433031    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433034    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433036    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433039    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433041    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433044    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433046    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433048    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433052    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433055    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433057    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433060    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433062    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:59:21.433648 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433065    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433067    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433070    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433072    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433076    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433080    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433085    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433088    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433091    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433094    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433097    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433099    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433102    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433104    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433107    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433109    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433112    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433115    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433118    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:59:21.434105 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433120    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433123    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433125    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433128    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433130    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433132    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433135    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433137    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433140    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433143    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433145    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433148    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433151    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433154    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433156    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433159    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433161    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433164    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433167    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433170    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:59:21.434537 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433172    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433175    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433177    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433179    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433182    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433184    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433186    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433189    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433191    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433195    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433199    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433203    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433206    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433208    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433211    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433887    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433893    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433896    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433899    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433902    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:59:21.435013 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433905    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433908    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433910    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433913    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433916    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433918    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433921    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433923    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433925    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433928    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433930    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433933    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433936    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433954    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433957    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433959    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433961    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433964    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433966    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433969    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:59:21.435466 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433971    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433973    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433976    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433978    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433981    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433984    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433986    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433988    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433990    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433993    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433995    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.433998    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434002    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434005    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434007    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434010    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434012    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434014    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434017    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434020    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:59:21.435958 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434022    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434024    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434027    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434029    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434032    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434034    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434037    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434039    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434042    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434044    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434046    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434050    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434053    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434056    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434059    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434061    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434064    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434067    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434069    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434071    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:59:21.436437 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434074    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434076    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434079    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434082    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434085    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434088    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434090    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434092    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434095    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434097    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434099    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434102    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434106    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434110    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434113    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434115    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434118    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434121    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434125    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:59:21.436914 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434129    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434131    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434134    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434137    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434139    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434141    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434144    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434147    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434149    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434151    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434154    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434156    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434158    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434161    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434163    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434540    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434545    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434547    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434550    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:59:21.437379 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434555    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434558    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434561    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434564    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434567    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434569    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434572    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434574    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434577    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434579    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434582    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434584    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434587    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434589    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434591    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434594    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434597    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434599    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434601    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434604    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:59:21.437822 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434606    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434609    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434611    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434614    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434616    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434618    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434621    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434623    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434625    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434628    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434631    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434633    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434636    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434638    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434641    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434643    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434645    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434648    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434650    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434652    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:59:21.438322 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434655    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434657    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434659    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434662    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434664    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434667    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434670    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434672    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434674    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434676    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434679    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434681    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434687    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434689    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434691    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434694    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434697    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434699    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434702    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434704    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:59:21.438783 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434706    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434709    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434711    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434713    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434715    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434718    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434720    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434722    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434725    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434727    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434730    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434732    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434734    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434737    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434739    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434741    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434744    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434746    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434749    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434752    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:59:21.439366 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434754    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434756    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434759    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434764    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434768    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434771    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434774    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434776    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434780    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434783    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434785    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434788    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434791    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434794    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.434796    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434870    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434877    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434883    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434887    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434891    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 17:59:21.439872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434894    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434898    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434902    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434905    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434908    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434910    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434913    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434916    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434919    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434922    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434925    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434928    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434931    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434935    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434951    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434955    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434959    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434962    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434966    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434969    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434973    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434976    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434979    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434982    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 17:59:21.440363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434984    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434987    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434990    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434994    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.434997    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435000    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435003    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435005    2562 flags.go:64] FLAG: --enable-server="true"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435008    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435012    2562 flags.go:64] FLAG: --event-burst="100"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435015    2562 flags.go:64] FLAG: --event-qps="50"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435017    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435021    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435024    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435028    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435030    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435034    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435037    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435039    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435042    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435045    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435048    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435051    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435053    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435057    2562 flags.go:64] FLAG: --feature-gates=""
Jul 23 17:59:21.441051 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435061    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435063    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435066    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435070    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435073    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435076    2562 flags.go:64] FLAG: --help="false"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435078    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435081    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435084    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435086    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435090    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435093    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435095    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435098    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435101    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435103    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435106    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435108    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435111    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435114    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435116    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435119    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435122    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435125    2562 flags.go:64] FLAG: --lock-file=""
Jul 23 17:59:21.442160 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435127    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435130    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435132    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435137    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435141    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435143    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435146    2562 flags.go:64] FLAG: --logging-format="text"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435150    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435153    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435155    2562 flags.go:64] FLAG: --manifest-url=""
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435158    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435163    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435166    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435169    2562 flags.go:64] FLAG: --max-pods="110"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435172    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435175    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435177    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435180    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435182    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435185    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435188    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435195    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435198    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435200    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 17:59:21.443151 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435203    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435206    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435210    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435213    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435216    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435218    2562 flags.go:64] FLAG: --port="10250"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435221    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435224    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-058ebe35efe2edd75"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435227    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435229    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435232    2562 flags.go:64] FLAG: --register-node="true"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435235    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435238    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435241    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435244    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435247    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435253    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435256    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435259    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435262    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435266    2562 flags.go:64] FLAG: --runonce="false"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435268    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435271    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435274    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435277    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 17:59:21.444195 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435279    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435282    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435285    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435287    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435290    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435292    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435295    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435297    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435300    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435303    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435305    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435331    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435334    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435337    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435341    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435344    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435346    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435349    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435352    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435354    2562 flags.go:64] FLAG: --v="2"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435358    2562 flags.go:64] FLAG: --version="false"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435363    2562 flags.go:64] FLAG: --vmodule=""
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435367    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435370    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435485    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:59:21.444971 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435488    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435491    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435495    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435498    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435501    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435503    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435505    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435508    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435510    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435513    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435515    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435517    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435519    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435522    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435524    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435526    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435529    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435531    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435533    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:59:21.446083 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435536    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435539    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435543    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435545    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435548    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435551    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435554    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435556    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435559    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435561    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435566    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435568    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435571    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435574    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435576    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435580    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435583    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435587    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435589    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:59:21.446670 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435592    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435595    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435597    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435600    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435602    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435605    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435607    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435609    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435612    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435614    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435617    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435619    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435622    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435624    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435626    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435628    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435631    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435633    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435635    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435638    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:59:21.447321 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435640    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435642    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435645    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435648    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435651    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435653    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435655    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435658    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435661    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435664    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435667    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435670    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435673    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435676    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435678    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435681    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435683    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435686    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435689    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435691    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:59:21.447815 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435693    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435696    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435698    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435701    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435703    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435705    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435707    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435710    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435712    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435714    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435716    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435719    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435721    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435724    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435726    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435731    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435733    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435736    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435738    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:59:21.448454 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.435741    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.435750    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.443050    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.443072    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.443100    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.443109    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443188    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443196    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443201    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443205    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443210    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443214    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443218    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443222    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443227    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:59:21.448978 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443231    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443235    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443239    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443243    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443247    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443251    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443256    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443260    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443265    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443271    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443275    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443280    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443284    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443288    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443292    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443296    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443300    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443304    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443309    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443313    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:59:21.449355 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443319    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443324    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443328    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443332    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443336    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443340    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443344    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443348    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443352    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443356    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443360    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443364    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443368    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443372    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443377    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443380    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443385    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443389    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443393    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443397    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:59:21.449836 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443401    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443405    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443409    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443413    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443417    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443422    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443426    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443430    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443434    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443438    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443443    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443447    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443452    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443457    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443461    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443464    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443469    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443473    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443477    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443481    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:59:21.450609 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443485    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443489    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443493    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443497    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443503    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443508    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443512    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443517    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443522    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443526    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443531    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443535    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443539    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443543    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443547    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443552    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443559    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443563    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443568    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:59:21.451392 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443572    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443576    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443581    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443585    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443590    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443594    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443599    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443604    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443609    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443614    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443618    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.443627    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443801    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443808    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443813    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:59:21.452002 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443818    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443822    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443826    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443830    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443834    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443838    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443843    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443847    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443851    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443855    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443859    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443863    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443867    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443871    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443875    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443879    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443883    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443887    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443891    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:59:21.452375 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443895    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443899    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443903    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443908    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443912    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443916    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443921    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443926    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443929    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443933    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443937    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443960    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443965    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443969    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443973    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443977    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443981    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443985    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443989    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443993    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:59:21.452930 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.443997    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444003    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444008    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444021    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444026    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444030    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444035    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444040    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444045    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444049    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444053    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444057    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444061    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444065    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444069    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444073    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444077    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444081    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444085    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:59:21.453445 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444090    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444094    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444098    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444102    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444106    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444110    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444113    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444117    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444121    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444125    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444129    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444133    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444137    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444141    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444144    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444148    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444152    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444156    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444160    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444164    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:59:21.454000 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444168    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444172    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444176    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444180    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444184    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444188    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444192    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444196    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444200    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444203    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444207    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444213    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444217    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444223    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444228    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444233    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444238    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:59:21.454473 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:21.444242    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.444251    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.444418    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.446931    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.448154    2562 server.go:1019] "Starting client certificate rotation"
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.448258    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.448323    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.450570    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 17:59:21.454870 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.454781    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 17:59:21.463120 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.463100    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 23 17:59:21.465655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.465635    2562 log.go:25] "Validated CRI v1 image API"
Jul 23 17:59:21.466594 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.466578    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 17:59:21.469303 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.469283    2562 fs.go:135] Filesystem UUIDs: map[1f4046f5-0992-48c7-8abe-4cac8a9181fa:/dev/nvme0n1p4 6227edae-43dc-4bdf-bcab-7a151f1af05c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 23 17:59:21.469370 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.469304    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 17:59:21.474136 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.474022    2562 manager.go:216] Machine: {Timestamp:2026-07-23 17:59:21.472974427 +0000 UTC m=+0.263131532 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200088 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e91fde18fe7c087a5c7e82476561 SystemUUID:ec28e91f-de18-fe7c-087a-5c7e82476561 BootID:b3784de1-accd-45d4-baf5-1dd9026d3106 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:55:7d:46:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:55:7d:46:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:d2:39:99:10:fe Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 17:59:21.474136 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.474125    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 17:59:21.474251 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.474208    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 17:59:21.475863 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.475848    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 17:59:21.476756 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.476733    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 17:59:21.476894 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.476760    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 17:59:21.477419 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.477409    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 17:59:21.477449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.477423    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 17:59:21.477449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.477430    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 17:59:21.478362 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.478350    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 17:59:21.479662 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.479652    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 17:59:21.483301 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483289    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 17:59:21.483410 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483401    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 17:59:21.483440 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483430    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 17:59:21.483471 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483440    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 17:59:21.483471 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483454    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 17:59:21.483471 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483463    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 23 17:59:21.483471 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483472    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 17:59:21.483866 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483854    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 17:59:21.483903 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.483877    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 17:59:21.485794 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.485780    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 17:59:21.488351 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.488332    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 17:59:21.488434 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.488367    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 17:59:21.489506 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489494    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 17:59:21.489542 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489512    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 17:59:21.489542 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489518    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 17:59:21.489542 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489523    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 17:59:21.489542 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489531    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 17:59:21.489542 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489537    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 17:59:21.489663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489545    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 17:59:21.489663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489550    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 17:59:21.489663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489557    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 17:59:21.489663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489562    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 17:59:21.489663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489576    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 17:59:21.489663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.489581    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 17:59:21.490138 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.490126    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 17:59:21.490172 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.490141    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 17:59:21.493039 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.493014    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-3.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 17:59:21.493039 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.493025    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 17:59:21.493178 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.493110    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 17:59:21.493604 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.493593    2562 server.go:1268] "Started kubelet"
Jul 23 17:59:21.493678 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.493652    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 17:59:21.493726 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.493681    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 17:59:21.493768 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.493731    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 17:59:21.494571 ip-10-0-130-3 systemd[1]: Started Kubernetes Kubelet.
Jul 23 17:59:21.495295 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.495275    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 17:59:21.495392 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.495380    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 17:59:21.498031 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.497276    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-3.ec2.internal.18c4fd079d70b0d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-3.ec2.internal,UID:ip-10-0-130-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-3.ec2.internal,},FirstTimestamp:2026-07-23 17:59:21.493565652 +0000 UTC m=+0.283722759,LastTimestamp:2026-07-23 17:59:21.493565652 +0000 UTC m=+0.283722759,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-3.ec2.internal,}"
Jul 23 17:59:21.498895 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.498873    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 17:59:21.499967 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.499933    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xg72"
Jul 23 17:59:21.500833 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.500813    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 17:59:21.500833 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.500823    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 17:59:21.501048 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.501030    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:21.501376 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501358    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 17:59:21.501376 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501377    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 17:59:21.501500 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501482    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 17:59:21.501500 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501491    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 17:59:21.501592 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501529    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 17:59:21.501641 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501613    2562 factory.go:55] Registering systemd factory
Jul 23 17:59:21.501690 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501646    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 23 17:59:21.501901 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501885    2562 factory.go:153] Registering CRI-O factory
Jul 23 17:59:21.501901 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501903    2562 factory.go:223] Registration of the crio container factory successfully
Jul 23 17:59:21.502065 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501961    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 17:59:21.502065 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501985    2562 factory.go:103] Registering Raw factory
Jul 23 17:59:21.502065 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.501997    2562 manager.go:1196] Started watching for new ooms in manager
Jul 23 17:59:21.502910 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.502898    2562 manager.go:349] Starting recovery of all containers
Jul 23 17:59:21.506316 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.506293    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xg72"
Jul 23 17:59:21.506494 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.506469    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 17:59:21.506612 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.506591    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 17:59:21.514066 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.513883    2562 manager.go:354] Recovery completed
Jul 23 17:59:21.519177 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.519164    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.521168 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521151    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.521240 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521188    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.521240 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521204    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.521741 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521728    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 17:59:21.521741 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521737    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.521867 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521739    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 17:59:21.521867 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.521798    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 17:59:21.522491 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.522474    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.522569 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.522501    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.522569 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.522511    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.523274 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.523197    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-3.ec2.internal.18c4fd079f15e95b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-3.ec2.internal,UID:ip-10-0-130-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-3.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-3.ec2.internal,},FirstTimestamp:2026-07-23 17:59:21.521170779 +0000 UTC m=+0.311327892,LastTimestamp:2026-07-23 17:59:21.521170779 +0000 UTC m=+0.311327892,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-3.ec2.internal,}"
Jul 23 17:59:21.523826 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.523813    2562 policy_none.go:49] "None policy: Start"
Jul 23 17:59:21.523872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.523829    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 17:59:21.523872 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.523839    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 17:59:21.525799 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.525788    2562 policy_none.go:47] "Start"
Jul 23 17:59:21.557549 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.557534    2562 manager.go:337] "Starting Device Plugin manager"
Jul 23 17:59:21.557656 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.557573    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 17:59:21.557656 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.557586    2562 server.go:87] "Starting device plugin registration server"
Jul 23 17:59:21.557849 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.557836    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 17:59:21.557902 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.557850    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 17:59:21.557982 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.557960    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 17:59:21.558071 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.558056    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 17:59:21.558118 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.558073    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 17:59:21.558581 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.558564    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 17:59:21.558650 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.558610    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:21.600157 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.600126    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 17:59:21.608039 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.601460    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 17:59:21.608039 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.601511    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 17:59:21.608039 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.601536    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 17:59:21.608039 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.601577    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 17:59:21.608039 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.603702    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:59:21.658615 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.658569    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.659577 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.659561    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.659642 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.659594    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.659642 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.659609    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.659642 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.659631    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.667323 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.667304    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.667390 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.667330    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-3.ec2.internal\": node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:21.676369 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.676346    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:21.701815 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.701758    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"]
Jul 23 17:59:21.701886 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.701847    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.703445 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.703430    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.703512 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.703459    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.703512 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.703473    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.705892 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.705880    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.706012 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.705999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.706056 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.706031    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.708328 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.708309    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.708428 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.708338    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.708428 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.708348    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.708428 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.708309    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.708428 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.708407    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.708428 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.708423    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.710697 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.710678    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.710768 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.710708    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:59:21.711564 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.711549    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:59:21.711630 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.711572    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:59:21.711630 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.711585    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:59:21.737669 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.737647    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-3.ec2.internal\" not found" node="ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.742115 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.742096    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-3.ec2.internal\" not found" node="ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.776670 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.776641    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:21.801916 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.801892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/638885469a6a1ee4b02e78d732b33fae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal\" (UID: \"638885469a6a1ee4b02e78d732b33fae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.802014 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.801918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/638885469a6a1ee4b02e78d732b33fae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal\" (UID: \"638885469a6a1ee4b02e78d732b33fae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.802014 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.801934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baee3903119d9c9ebdaea817602fe47e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-3.ec2.internal\" (UID: \"baee3903119d9c9ebdaea817602fe47e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.877512 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.877475    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:21.902842 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.902814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/638885469a6a1ee4b02e78d732b33fae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal\" (UID: \"638885469a6a1ee4b02e78d732b33fae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.902918 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.902854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/638885469a6a1ee4b02e78d732b33fae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal\" (UID: \"638885469a6a1ee4b02e78d732b33fae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.902918 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.902895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/638885469a6a1ee4b02e78d732b33fae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal\" (UID: \"638885469a6a1ee4b02e78d732b33fae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.902918 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.902912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baee3903119d9c9ebdaea817602fe47e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-3.ec2.internal\" (UID: \"baee3903119d9c9ebdaea817602fe47e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.903030 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.902936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baee3903119d9c9ebdaea817602fe47e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-3.ec2.internal\" (UID: \"baee3903119d9c9ebdaea817602fe47e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.903030 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:21.902983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/638885469a6a1ee4b02e78d732b33fae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal\" (UID: \"638885469a6a1ee4b02e78d732b33fae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:21.978167 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:21.978101    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.040795 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.040757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:22.044610 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.044596    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:22.078701 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.078670    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.179193 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.179155    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.279792 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.279709    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.380225 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.380191    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.447784 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.447740    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 17:59:22.448267 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.447910    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 17:59:22.481019 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.480988    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.514331 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.514304    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 17:54:21 +0000 UTC" deadline="2028-04-21 20:39:50.657060906 +0000 UTC"
Jul 23 17:59:22.514331 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.514330    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15314h40m28.142733465s"
Jul 23 17:59:22.581673 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.581600    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.624462 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:22.624423    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaee3903119d9c9ebdaea817602fe47e.slice/crio-3eef4217ad3f796607c253c263d7ebd6007d47a27fe282767ebd54c8cb64d3c1 WatchSource:0}: Error finding container 3eef4217ad3f796607c253c263d7ebd6007d47a27fe282767ebd54c8cb64d3c1: Status 404 returned error can't find the container with id 3eef4217ad3f796607c253c263d7ebd6007d47a27fe282767ebd54c8cb64d3c1
Jul 23 17:59:22.629425 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.629409    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:59:22.634478 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:22.634454    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638885469a6a1ee4b02e78d732b33fae.slice/crio-332b4b0aaf2bcc04c167f15a7e6b8ed9f634d669b81f85a2a383cfa16565ed44 WatchSource:0}: Error finding container 332b4b0aaf2bcc04c167f15a7e6b8ed9f634d669b81f85a2a383cfa16565ed44: Status 404 returned error can't find the container with id 332b4b0aaf2bcc04c167f15a7e6b8ed9f634d669b81f85a2a383cfa16565ed44
Jul 23 17:59:22.682246 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.682203    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.782822 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.782788    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.787811 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.787792    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:59:22.883934 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:22.883855    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-3.ec2.internal\" not found"
Jul 23 17:59:22.963816 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:22.963793    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:59:23.002274 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.002234    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:23.014754 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.014730    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 17:59:23.016217 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.016202    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal"
Jul 23 17:59:23.041004 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.040961    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:59:23.130316 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.130167    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 17:59:23.485258 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.485227    2562 apiserver.go:52] "Watching apiserver"
Jul 23 17:59:23.495799 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.495768    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 17:59:23.496896 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.496864    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hbfsq","kube-system/global-pull-secret-syncer-gdqpw","kube-system/konnectivity-agent-c789p","kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xjkjp","openshift-dns/node-resolver-9xbxt","openshift-image-registry/node-ca-lwdc5","openshift-multus/network-metrics-daemon-vd8gk","openshift-network-diagnostics/network-check-target-z2zwn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal","openshift-multus/multus-additional-cni-plugins-lqf6q","openshift-multus/multus-lxftc","openshift-network-operator/iptables-alerter-qfjkw"]
Jul 23 17:59:23.499591 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.499560    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.502489 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.501644    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 17:59:23.502489 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.501961    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 17:59:23.502489 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.502196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.502489 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.502263    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.502489 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.502374    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rw676\""
Jul 23 17:59:23.502489 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.502448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.505192 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.504612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.505192 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.505035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 17:59:23.505192 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.505049    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.505412 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.505260    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 17:59:23.505412 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.505310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.505412 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.505260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7dxcl\""
Jul 23 17:59:23.506922 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.506900    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 17:59:23.507132 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.507086    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.507672 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.507487    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2cv5h\""
Jul 23 17:59:23.507672 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.507592    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 17:59:23.509254 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.509081    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.509448 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.509427    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 17:59:23.509517 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.509478    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.509610 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.509393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jcpl2\""
Jul 23 17:59:23.510127 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.510100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.511724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.511823 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6cm\" (UniqueName: \"kubernetes.io/projected/d3bf21c2-367d-4495-9e6e-f4debd467e6a-kube-api-access-rx6cm\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.511823 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-cni-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.511823 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-socket-dir-parent\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.511823 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-k8s-cni-cncf-io\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-cni-bin\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-etc-kubernetes\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-cnibin\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.511961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c232cece-5ed5-4463-b121-e393c4b88ff1-cni-binary-copy\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-hostroot\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512104 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.512614 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-system-cni-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512679 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-os-release\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512679 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-netns\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-cni-multus\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512410    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.512783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-daemon-config\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.512783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.512783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512752    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-device-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.512783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-kubelet\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-conf-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-multus-certs\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbjj\" (UniqueName: \"kubernetes.io/projected/c232cece-5ed5-4463-b121-e393c4b88ff1-kube-api-access-7kbjj\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512403    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6l4xb\""
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512466    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512439    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.513105 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.512453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.514902 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.514866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:23.515036 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.515015    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:23.515122 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.515096    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.515327 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.515307    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gh5p5\""
Jul 23 17:59:23.515327 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.515322    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.517258 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.517240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.519369 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.519348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.519466 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.519446    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kddb5\""
Jul 23 17:59:23.519528 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.519509    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.519810 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.519791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.519885 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.519864    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:23.520456 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.520400    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 17:59:23.522406 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.522388    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.524834 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.524811    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 17:59:23.524921 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.524839    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 17:59:23.524921 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.524877    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 17:59:23.525274 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.525255    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kb9x4\""
Jul 23 17:59:23.525624 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.525561    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 17:59:23.525713 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.525645    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 17:59:23.529860 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.526562    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 17:59:23.529860 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.527001    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:23.529860 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.527112    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:23.530743 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.530726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.533413 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.532899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 17:59:23.533413 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.533257    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x2d9n\""
Jul 23 17:59:23.533413 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.533315    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 17:59:23.536797 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.536774    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wmcr2"
Jul 23 17:59:23.545663 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.545640    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wmcr2"
Jul 23 17:59:23.603398 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.603368    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 17:59:23.606522 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.606469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal" event={"ID":"baee3903119d9c9ebdaea817602fe47e","Type":"ContainerStarted","Data":"3eef4217ad3f796607c253c263d7ebd6007d47a27fe282767ebd54c8cb64d3c1"}
Jul 23 17:59:23.607893 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.607866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal" event={"ID":"638885469a6a1ee4b02e78d732b33fae","Type":"ContainerStarted","Data":"332b4b0aaf2bcc04c167f15a7e6b8ed9f634d669b81f85a2a383cfa16565ed44"}
Jul 23 17:59:23.613205 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-system-cni-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-os-release\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-cni-multus\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-cni-multus\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-system-cni-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-os-release\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da29655f-7bd7-490f-926b-c40c5353dffc-host\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.613390 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-kubernetes\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-cni-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-etc-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-lib-modules\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhlc\" (UniqueName: \"kubernetes.io/projected/94942e6d-c9a9-4fb5-bac0-4aa81845e979-kube-api-access-gvhlc\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fcbf1c01-1735-4764-9986-a750528b484e-dbus\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-hostroot\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-os-release\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysctl-conf\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-host\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-cni-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-tuned\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.613776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-hostroot\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613775    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-multus-certs\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613874    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbxc\" (UniqueName: \"kubernetes.io/projected/da29655f-7bd7-490f-926b-c40c5353dffc-kube-api-access-xvbxc\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-run-netns\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-var-lib-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-multus-certs\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.613999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-kubelet\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-conf-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-systemd-units\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-systemd\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-kubelet\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.614402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-conf-dir\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-modprobe-d\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6cm\" (UniqueName: \"kubernetes.io/projected/d3bf21c2-367d-4495-9e6e-f4debd467e6a-kube-api-access-rx6cm\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-socket-dir-parent\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-etc-kubernetes\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da29655f-7bd7-490f-926b-c40c5353dffc-serviceca\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-ovn\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-systemd\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-kubelet\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysconfig\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-sys\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94942e6d-c9a9-4fb5-bac0-4aa81845e979-hosts-file\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-node-log\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.614964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcqg\" (UniqueName: \"kubernetes.io/projected/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-kube-api-access-6lcqg\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ea73276-b4e6-46cc-bf41-acb16a94142a-host-slash\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-run\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-log-socket\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovnkube-script-lib\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxh6q\" (UniqueName: \"kubernetes.io/projected/067b5e5d-0bd1-41cd-b076-7118f4472717-kube-api-access-nxh6q\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhsm\" (UniqueName: \"kubernetes.io/projected/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-kube-api-access-pqhsm\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94942e6d-c9a9-4fb5-bac0-4aa81845e979-tmp-dir\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fcbf1c01-1735-4764-9986-a750528b484e-kubelet-config\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-cnibin\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfm6q\" (UniqueName: \"kubernetes.io/projected/4ea73276-b4e6-46cc-bf41-acb16a94142a-kube-api-access-zfm6q\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovnkube-config\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-system-cni-dir\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.615724 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-netns\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-daemon-config\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9f3a7bc-def3-4b79-b099-abf23b7f57a5-konnectivity-ca\") pod \"konnectivity-agent-c789p\" (UID: \"a9f3a7bc-def3-4b79-b099-abf23b7f57a5\") " pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-slash\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-cni-bin\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-cni-netd\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-device-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.614986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbjj\" (UniqueName: \"kubernetes.io/projected/c232cece-5ed5-4463-b121-e393c4b88ff1-kube-api-access-7kbjj\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9f3a7bc-def3-4b79-b099-abf23b7f57a5-agent-certs\") pod \"konnectivity-agent-c789p\" (UID: \"a9f3a7bc-def3-4b79-b099-abf23b7f57a5\") " pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615459    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-netns\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-socket-dir-parent\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysctl-d\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-var-lib-kubelet\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-tmp\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:23.616858 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-k8s-cni-cncf-io\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-run-k8s-cni-cncf-io\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-cni-bin\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.615989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovn-node-metrics-cert\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3bf21c2-367d-4495-9e6e-f4debd467e6a-device-dir\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78mz\" (UniqueName: \"kubernetes.io/projected/197e0944-75f2-4c47-8ed0-53194766062d-kube-api-access-m78mz\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ea73276-b4e6-46cc-bf41-acb16a94142a-iptables-alerter-script\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-cnibin\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c232cece-5ed5-4463-b121-e393c4b88ff1-cni-binary-copy\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-env-overrides\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-host-var-lib-cni-bin\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c232cece-5ed5-4463-b121-e393c4b88ff1-multus-daemon-config\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-cnibin\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c232cece-5ed5-4463-b121-e393c4b88ff1-etc-kubernetes\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.617449 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.616765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c232cece-5ed5-4463-b121-e393c4b88ff1-cni-binary-copy\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.622155 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.622132    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 17:59:23.625856 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.625835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6cm\" (UniqueName: \"kubernetes.io/projected/d3bf21c2-367d-4495-9e6e-f4debd467e6a-kube-api-access-rx6cm\") pod \"aws-ebs-csi-driver-node-thfl9\" (UID: \"d3bf21c2-367d-4495-9e6e-f4debd467e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.625986 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.625860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbjj\" (UniqueName: \"kubernetes.io/projected/c232cece-5ed5-4463-b121-e393c4b88ff1-kube-api-access-7kbjj\") pod \"multus-lxftc\" (UID: \"c232cece-5ed5-4463-b121-e393c4b88ff1\") " pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.717373 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-etc-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.717373 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-lib-modules\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhlc\" (UniqueName: \"kubernetes.io/projected/94942e6d-c9a9-4fb5-bac0-4aa81845e979-kube-api-access-gvhlc\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-etc-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-lib-modules\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fcbf1c01-1735-4764-9986-a750528b484e-dbus\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-os-release\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysctl-conf\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.717696 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-host\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fcbf1c01-1735-4764-9986-a750528b484e-dbus\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-tuned\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-os-release\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbxc\" (UniqueName: \"kubernetes.io/projected/da29655f-7bd7-490f-926b-c40c5353dffc-kube-api-access-xvbxc\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717760    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-host\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-run-netns\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717829    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-run-netns\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717829    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysctl-conf\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-var-lib-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-var-lib-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-systemd-units\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.717990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-systemd\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718067 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-systemd-units\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-systemd\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-modprobe-d\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-openvswitch\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da29655f-7bd7-490f-926b-c40c5353dffc-serviceca\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-ovn\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-systemd\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-run-ovn\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-systemd\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-modprobe-d\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-kubelet\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718373    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysconfig\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-sys\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-kubelet\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-sys\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.718933 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94942e6d-c9a9-4fb5-bac0-4aa81845e979-hosts-file\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysconfig\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-node-log\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.718581    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcqg\" (UniqueName: \"kubernetes.io/projected/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-kube-api-access-6lcqg\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ea73276-b4e6-46cc-bf41-acb16a94142a-host-slash\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-run\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-log-socket\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94942e6d-c9a9-4fb5-bac0-4aa81845e979-hosts-file\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovnkube-script-lib\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/067b5e5d-0bd1-41cd-b076-7118f4472717-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718779    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxh6q\" (UniqueName: \"kubernetes.io/projected/067b5e5d-0bd1-41cd-b076-7118f4472717-kube-api-access-nxh6q\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-node-log\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.719800 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhsm\" (UniqueName: \"kubernetes.io/projected/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-kube-api-access-pqhsm\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94942e6d-c9a9-4fb5-bac0-4aa81845e979-tmp-dir\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fcbf1c01-1735-4764-9986-a750528b484e-kubelet-config\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da29655f-7bd7-490f-926b-c40c5353dffc-serviceca\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ea73276-b4e6-46cc-bf41-acb16a94142a-host-slash\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-cnibin\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.718925    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 17:59:24.218889453 +0000 UTC m=+3.009046548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fcbf1c01-1735-4764-9986-a750528b484e-kubelet-config\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfm6q\" (UniqueName: \"kubernetes.io/projected/4ea73276-b4e6-46cc-bf41-acb16a94142a-kube-api-access-zfm6q\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovnkube-config\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-system-cni-dir\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9f3a7bc-def3-4b79-b099-abf23b7f57a5-konnectivity-ca\") pod \"konnectivity-agent-c789p\" (UID: \"a9f3a7bc-def3-4b79-b099-abf23b7f57a5\") " pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-log-socket\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.718867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-run\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.720841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-slash\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-cni-bin\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-cni-netd\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-cnibin\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719169    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-slash\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9f3a7bc-def3-4b79-b099-abf23b7f57a5-agent-certs\") pod \"konnectivity-agent-c789p\" (UID: \"a9f3a7bc-def3-4b79-b099-abf23b7f57a5\") " pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysctl-d\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/067b5e5d-0bd1-41cd-b076-7118f4472717-system-cni-dir\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-cni-bin\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-var-lib-kubelet\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94942e6d-c9a9-4fb5-bac0-4aa81845e979-tmp-dir\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-tmp\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovnkube-script-lib\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-host-cni-netd\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-var-lib-kubelet\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-sysctl-d\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.721655 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovn-node-metrics-cert\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78mz\" (UniqueName: \"kubernetes.io/projected/197e0944-75f2-4c47-8ed0-53194766062d-kube-api-access-m78mz\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ea73276-b4e6-46cc-bf41-acb16a94142a-iptables-alerter-script\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-env-overrides\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da29655f-7bd7-490f-926b-c40c5353dffc-host\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-kubernetes\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovnkube-config\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.719765    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.719813    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:24.219798005 +0000 UTC m=+3.009955105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9f3a7bc-def3-4b79-b099-abf23b7f57a5-konnectivity-ca\") pod \"konnectivity-agent-c789p\" (UID: \"a9f3a7bc-def3-4b79-b099-abf23b7f57a5\") " pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-kubernetes\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.719928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da29655f-7bd7-490f-926b-c40c5353dffc-host\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.720309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ea73276-b4e6-46cc-bf41-acb16a94142a-iptables-alerter-script\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.720419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-env-overrides\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.722452 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.722191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-tmp\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.722997 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.722663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9f3a7bc-def3-4b79-b099-abf23b7f57a5-agent-certs\") pod \"konnectivity-agent-c789p\" (UID: \"a9f3a7bc-def3-4b79-b099-abf23b7f57a5\") " pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.723205 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.723184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-ovn-node-metrics-cert\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.723471 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.723383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-etc-tuned\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.726437 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.726292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbxc\" (UniqueName: \"kubernetes.io/projected/da29655f-7bd7-490f-926b-c40c5353dffc-kube-api-access-xvbxc\") pod \"node-ca-lwdc5\" (UID: \"da29655f-7bd7-490f-926b-c40c5353dffc\") " pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.726437 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.726360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhlc\" (UniqueName: \"kubernetes.io/projected/94942e6d-c9a9-4fb5-bac0-4aa81845e979-kube-api-access-gvhlc\") pod \"node-resolver-9xbxt\" (UID: \"94942e6d-c9a9-4fb5-bac0-4aa81845e979\") " pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.726652 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.726625    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:23.726652 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.726642    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:23.726652 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.726654    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:23.726833 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:23.726710    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:24.226694703 +0000 UTC m=+3.016851812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:23.728446 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.728422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxh6q\" (UniqueName: \"kubernetes.io/projected/067b5e5d-0bd1-41cd-b076-7118f4472717-kube-api-access-nxh6q\") pod \"multus-additional-cni-plugins-lqf6q\" (UID: \"067b5e5d-0bd1-41cd-b076-7118f4472717\") " pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:23.729130 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.729070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfm6q\" (UniqueName: \"kubernetes.io/projected/4ea73276-b4e6-46cc-bf41-acb16a94142a-kube-api-access-zfm6q\") pod \"iptables-alerter-qfjkw\" (UID: \"4ea73276-b4e6-46cc-bf41-acb16a94142a\") " pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.729703 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.729494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcqg\" (UniqueName: \"kubernetes.io/projected/5bfa618f-330c-4aa6-9b3a-9aac777c1cf1-kube-api-access-6lcqg\") pod \"ovnkube-node-hbfsq\" (UID: \"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.729986 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.729967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhsm\" (UniqueName: \"kubernetes.io/projected/0169b9fc-cb87-4143-aaf3-9ed4a4e00946-kube-api-access-pqhsm\") pod \"tuned-xjkjp\" (UID: \"0169b9fc-cb87-4143-aaf3-9ed4a4e00946\") " pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.730771 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.730750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78mz\" (UniqueName: \"kubernetes.io/projected/197e0944-75f2-4c47-8ed0-53194766062d-kube-api-access-m78mz\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:23.812433 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.812346    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9"
Jul 23 17:59:23.815799 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.815771    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:59:23.820901 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.820875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxftc"
Jul 23 17:59:23.836114 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.836092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:23.844084 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.844055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qfjkw"
Jul 23 17:59:23.853064 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.853044    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xjkjp"
Jul 23 17:59:23.861998 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.861973    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9xbxt"
Jul 23 17:59:23.868899 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.868871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lwdc5"
Jul 23 17:59:23.876834 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.876810    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:23.884905 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:23.884882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lqf6q"
Jul 23 17:59:24.224171 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:24.224099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:24.224171 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:24.224168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:24.224396 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.224292    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:24.224396 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.224293    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:24.224396 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.224358    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:25.224340533 +0000 UTC m=+4.014497646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:24.224396 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.224377    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 17:59:25.224367443 +0000 UTC m=+4.014524552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:24.324764 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:24.324730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:24.324926 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.324909    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:24.325009 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.324930    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:24.325009 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.324940    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:24.325087 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.325018    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:25.325002576 +0000 UTC m=+4.115159671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:24.549352 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:24.549258    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:54:23 +0000 UTC" deadline="2028-03-10 16:39:18.340168177 +0000 UTC"
Jul 23 17:59:24.549352 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:24.549291    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14302h39m53.790880283s"
Jul 23 17:59:24.601841 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:24.601791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:24.602044 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:24.601943    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:24.727284 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.727259    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067b5e5d_0bd1_41cd_b076_7118f4472717.slice/crio-9db08828e22cdea77441a80eda83f2068de09e96b465cf36666ac5720c26ee55 WatchSource:0}: Error finding container 9db08828e22cdea77441a80eda83f2068de09e96b465cf36666ac5720c26ee55: Status 404 returned error can't find the container with id 9db08828e22cdea77441a80eda83f2068de09e96b465cf36666ac5720c26ee55
Jul 23 17:59:24.729222 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.728932    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f3a7bc_def3_4b79_b099_abf23b7f57a5.slice/crio-60ff5df4a0b5172b4ae4f1c3a937d8606d7008a9b8504709334b6e544a3af90a WatchSource:0}: Error finding container 60ff5df4a0b5172b4ae4f1c3a937d8606d7008a9b8504709334b6e544a3af90a: Status 404 returned error can't find the container with id 60ff5df4a0b5172b4ae4f1c3a937d8606d7008a9b8504709334b6e544a3af90a
Jul 23 17:59:24.730695 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.730667    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94942e6d_c9a9_4fb5_bac0_4aa81845e979.slice/crio-f2a4d5395d69baddb58e7870d90c58f2ef8c21f0167150d425894926e61bb513 WatchSource:0}: Error finding container f2a4d5395d69baddb58e7870d90c58f2ef8c21f0167150d425894926e61bb513: Status 404 returned error can't find the container with id f2a4d5395d69baddb58e7870d90c58f2ef8c21f0167150d425894926e61bb513
Jul 23 17:59:24.731835 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.731748    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfa618f_330c_4aa6_9b3a_9aac777c1cf1.slice/crio-9bec4b776722006540f7866e1b6e3d1c8ebab37b93702e4ca0f629bffb46f064 WatchSource:0}: Error finding container 9bec4b776722006540f7866e1b6e3d1c8ebab37b93702e4ca0f629bffb46f064: Status 404 returned error can't find the container with id 9bec4b776722006540f7866e1b6e3d1c8ebab37b93702e4ca0f629bffb46f064
Jul 23 17:59:24.737668 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.735639    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda29655f_7bd7_490f_926b_c40c5353dffc.slice/crio-559aede9b60feae4f73a4c657bd5a5540d216dfae68adbc785455d225a725cf4 WatchSource:0}: Error finding container 559aede9b60feae4f73a4c657bd5a5540d216dfae68adbc785455d225a725cf4: Status 404 returned error can't find the container with id 559aede9b60feae4f73a4c657bd5a5540d216dfae68adbc785455d225a725cf4
Jul 23 17:59:24.737668 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.736621    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0169b9fc_cb87_4143_aaf3_9ed4a4e00946.slice/crio-680c86223b94628db5b16fee5d06fc385a70f6041aa62ee3e4a03825ccdefb30 WatchSource:0}: Error finding container 680c86223b94628db5b16fee5d06fc385a70f6041aa62ee3e4a03825ccdefb30: Status 404 returned error can't find the container with id 680c86223b94628db5b16fee5d06fc385a70f6041aa62ee3e4a03825ccdefb30
Jul 23 17:59:24.738763 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.738688    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc232cece_5ed5_4463_b121_e393c4b88ff1.slice/crio-a2068bb95ccda995865fb0e367b1473d2c9d22acf9541c8f3ca6f736447a8437 WatchSource:0}: Error finding container a2068bb95ccda995865fb0e367b1473d2c9d22acf9541c8f3ca6f736447a8437: Status 404 returned error can't find the container with id a2068bb95ccda995865fb0e367b1473d2c9d22acf9541c8f3ca6f736447a8437
Jul 23 17:59:24.739705 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.739674    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea73276_b4e6_46cc_bf41_acb16a94142a.slice/crio-224649a4258fd0f3797a7a86a658bc3b1d405238ebedb4217e45490f7274dc91 WatchSource:0}: Error finding container 224649a4258fd0f3797a7a86a658bc3b1d405238ebedb4217e45490f7274dc91: Status 404 returned error can't find the container with id 224649a4258fd0f3797a7a86a658bc3b1d405238ebedb4217e45490f7274dc91
Jul 23 17:59:24.742131 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:24.742110    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bf21c2_367d_4495_9e6e_f4debd467e6a.slice/crio-97ad7f38f72ea6df7fd4c08dd5cf15a65cdb0b41e294220ee28ec0ee2527ddf9 WatchSource:0}: Error finding container 97ad7f38f72ea6df7fd4c08dd5cf15a65cdb0b41e294220ee28ec0ee2527ddf9: Status 404 returned error can't find the container with id 97ad7f38f72ea6df7fd4c08dd5cf15a65cdb0b41e294220ee28ec0ee2527ddf9
Jul 23 17:59:25.232420 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.232083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:25.232589 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.232436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:25.232589 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.232558    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:25.232697 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.232619    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:27.232600202 +0000 UTC m=+6.022757315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:25.242924 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.242198    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:25.242924 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.242393    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 17:59:27.242370195 +0000 UTC m=+6.032527300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:25.333576 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.332913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:25.333576 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.333093    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:25.333576 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.333113    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:25.333576 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.333128    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:25.333576 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.333186    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:27.333165142 +0000 UTC m=+6.123322256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:25.550466 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.550345    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:54:23 +0000 UTC" deadline="2028-02-21 09:50:17.294008361 +0000 UTC"
Jul 23 17:59:25.550466 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.550416    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13863h50m51.743597841s"
Jul 23 17:59:25.604472 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.604437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:25.604641 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.604572    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:25.605116 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.604990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:25.605286 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:25.605125    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:25.641925 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.641832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9" event={"ID":"d3bf21c2-367d-4495-9e6e-f4debd467e6a","Type":"ContainerStarted","Data":"97ad7f38f72ea6df7fd4c08dd5cf15a65cdb0b41e294220ee28ec0ee2527ddf9"}
Jul 23 17:59:25.649919 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.649880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lwdc5" event={"ID":"da29655f-7bd7-490f-926b-c40c5353dffc","Type":"ContainerStarted","Data":"559aede9b60feae4f73a4c657bd5a5540d216dfae68adbc785455d225a725cf4"}
Jul 23 17:59:25.658419 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.658383    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c789p" event={"ID":"a9f3a7bc-def3-4b79-b099-abf23b7f57a5","Type":"ContainerStarted","Data":"60ff5df4a0b5172b4ae4f1c3a937d8606d7008a9b8504709334b6e544a3af90a"}
Jul 23 17:59:25.668215 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.668177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9xbxt" event={"ID":"94942e6d-c9a9-4fb5-bac0-4aa81845e979","Type":"ContainerStarted","Data":"f2a4d5395d69baddb58e7870d90c58f2ef8c21f0167150d425894926e61bb513"}
Jul 23 17:59:25.674233 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.674202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerStarted","Data":"9db08828e22cdea77441a80eda83f2068de09e96b465cf36666ac5720c26ee55"}
Jul 23 17:59:25.685572 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.685540    2562 generic.go:356] "Generic (PLEG): container finished" podID="638885469a6a1ee4b02e78d732b33fae" containerID="1cd8c4abb0eefc5fd4f017fad7683d36679374b894999fe6b37dac7f3027b7ea" exitCode=0
Jul 23 17:59:25.685719 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.685631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal" event={"ID":"638885469a6a1ee4b02e78d732b33fae","Type":"ContainerDied","Data":"1cd8c4abb0eefc5fd4f017fad7683d36679374b894999fe6b37dac7f3027b7ea"}
Jul 23 17:59:25.732368 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.731848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal" event={"ID":"baee3903119d9c9ebdaea817602fe47e","Type":"ContainerStarted","Data":"0bab25124fb3b1cc9d840286c8f654deba251b4d36cacad55e941b1da72b1ab2"}
Jul 23 17:59:25.744791 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.744749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qfjkw" event={"ID":"4ea73276-b4e6-46cc-bf41-acb16a94142a","Type":"ContainerStarted","Data":"224649a4258fd0f3797a7a86a658bc3b1d405238ebedb4217e45490f7274dc91"}
Jul 23 17:59:25.749108 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.748909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xjkjp" event={"ID":"0169b9fc-cb87-4143-aaf3-9ed4a4e00946","Type":"ContainerStarted","Data":"680c86223b94628db5b16fee5d06fc385a70f6041aa62ee3e4a03825ccdefb30"}
Jul 23 17:59:25.764388 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.764122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxftc" event={"ID":"c232cece-5ed5-4463-b121-e393c4b88ff1","Type":"ContainerStarted","Data":"a2068bb95ccda995865fb0e367b1473d2c9d22acf9541c8f3ca6f736447a8437"}
Jul 23 17:59:25.771615 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:25.771575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"9bec4b776722006540f7866e1b6e3d1c8ebab37b93702e4ca0f629bffb46f064"}
Jul 23 17:59:26.602344 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:26.602308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:26.602849 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:26.602465    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:26.795248 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:26.795212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal" event={"ID":"638885469a6a1ee4b02e78d732b33fae","Type":"ContainerStarted","Data":"b6f906d4cb3844d4eca407e565e8000a6d05bdd92b3ac75b61a76f33860b029e"}
Jul 23 17:59:26.812488 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:26.812427    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-3.ec2.internal" podStartSLOduration=3.812407151 podStartE2EDuration="3.812407151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:26.811966269 +0000 UTC m=+5.602123389" watchObservedRunningTime="2026-07-23 17:59:26.812407151 +0000 UTC m=+5.602564271"
Jul 23 17:59:26.813297 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:26.813104    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-3.ec2.internal" podStartSLOduration=3.813089866 podStartE2EDuration="3.813089866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:25.748307412 +0000 UTC m=+4.538464531" watchObservedRunningTime="2026-07-23 17:59:26.813089866 +0000 UTC m=+5.603246986"
Jul 23 17:59:27.249568 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:27.249528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:27.249745 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:27.249602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:27.249796 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.249751    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:27.249847 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.249816    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 17:59:31.249798005 +0000 UTC m=+10.039955105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:27.250262 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.250241    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:27.250340 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.250296    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:31.250281434 +0000 UTC m=+10.040438534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:27.355042 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:27.350667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:27.355042 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.350855    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:27.355042 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.350878    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:27.355042 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.350892    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:27.355042 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.350958    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:31.350937879 +0000 UTC m=+10.141095008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:27.603660 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:27.603578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:27.604359 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.603689    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:27.604359 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:27.603739    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:27.604359 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:27.603783    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:28.602726 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:28.602684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:28.602909 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:28.602855    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:29.602099 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:29.602061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:29.602553 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:29.602201    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:29.602553 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:29.602282    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:29.602553 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:29.602410    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:30.602490 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:30.602448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:30.602879 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:30.602588    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:31.284424 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:31.284382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:31.284578 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:31.284453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:31.284637 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.284602    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:31.284675 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.284660    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 17:59:39.284643339 +0000 UTC m=+18.074800437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:31.284727 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.284719    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:31.284760 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.284749    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:39.284738226 +0000 UTC m=+18.074895325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:31.385860 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:31.385718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:31.386064 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.385916    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:31.386064 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.385955    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:31.386064 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.385969    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:31.386064 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.386024    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:39.386006964 +0000 UTC m=+18.176164057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:31.602801 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:31.602717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:31.603265 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.602837    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:31.603265 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:31.602891    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:31.603265 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:31.603021    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:32.602267 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:32.601839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:32.602267 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:32.601967    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:33.602781 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:33.602740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:33.603193 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:33.602785    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:33.603193 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:33.602856    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:33.603193 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:33.603009    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:34.602734 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:34.602696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:34.602909 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:34.602843    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:35.603207 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:35.602644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:35.603207 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:35.602643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:35.603207 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:35.602779    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:35.603207 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:35.602863    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:36.601853 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:36.601817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:36.602062 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:36.601974    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:37.601859 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:37.601824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:37.602262 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:37.601990    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:37.602262 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:37.602027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:37.602262 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:37.602140    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:38.602698 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:38.602663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:38.603223 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:38.602788    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:39.350811 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:39.350779    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:39.350954 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:39.350825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:39.351011 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.350936    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:39.351011 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.350936    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:39.351070 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.351015    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:55.350997963 +0000 UTC m=+34.141155057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:39.351070 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.351032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 17:59:55.351024143 +0000 UTC m=+34.141181237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:39.451961 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:39.451910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:39.452131 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.452071    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:39.452131 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.452086    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:39.452131 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.452096    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:39.452234 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.452151    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 17:59:55.452135727 +0000 UTC m=+34.242292836 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:39.602149 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:39.602067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:39.602299 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.602199    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:39.602299 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:39.602226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:39.602413 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:39.602327    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:40.602114 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:40.602071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:40.602567 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:40.602229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:41.602570 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:41.602536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:41.603038 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:41.602646    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:41.603038 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:41.602702    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:41.603038 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:41.602797    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:42.601787 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:42.601747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:42.601991 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:42.601889    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:43.602264 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:43.601980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:43.602264 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:43.602096    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:43.602264 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:43.602166    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:43.602264 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:43.602238    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:43.843687 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:43.843518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9" event={"ID":"d3bf21c2-367d-4495-9e6e-f4debd467e6a","Type":"ContainerStarted","Data":"064ab6486f8d2352a19431be558b76dce6f08608ad344a2fe606cee1da426348"}
Jul 23 17:59:43.844964 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:43.844924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c789p" event={"ID":"a9f3a7bc-def3-4b79-b099-abf23b7f57a5","Type":"ContainerStarted","Data":"cb57f6e42021a0adcf47c216175cba3160b42c6d7bc406cf1dc47e6005ca3094"}
Jul 23 17:59:43.862564 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:43.862527    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c789p" podStartSLOduration=3.05219904 podStartE2EDuration="21.862512582s" totalImagesPullingTime="18.810313542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:43.861956995 +0000 UTC m=+22.652114106" watchObservedRunningTime="2026-07-23 17:59:43.862512582 +0000 UTC m=+22.652669696"
Jul 23 17:59:44.348241 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.348064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:44.348581 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.348568    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:44.601980 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.601882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:44.602126 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:44.602039    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:44.848927 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.848880    2562 generic.go:356] "Generic (PLEG): container finished" podID="067b5e5d-0bd1-41cd-b076-7118f4472717" containerID="7a64629bb6a19315a089f140b31d52bbf4c8d88b98242bf2de71bc9d9323c927" exitCode=0
Jul 23 17:59:44.849890 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.848975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerDied","Data":"7a64629bb6a19315a089f140b31d52bbf4c8d88b98242bf2de71bc9d9323c927"}
Jul 23 17:59:44.850319 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.850296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xjkjp" event={"ID":"0169b9fc-cb87-4143-aaf3-9ed4a4e00946","Type":"ContainerStarted","Data":"c4fb2234667f482aa4be8feffea64e7f59ea22b9e49531b7b6a1c8fd9cd59ce6"}
Jul 23 17:59:44.851783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.851753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxftc" event={"ID":"c232cece-5ed5-4463-b121-e393c4b88ff1","Type":"ContainerStarted","Data":"40acc374bdf1e5daaf7cd54c61fc53e2b96aaa142a07e5dee91d1e111cfc3904"}
Jul 23 17:59:44.854432 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854384    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 17:59:44.854738 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854715    2562 generic.go:356] "Generic (PLEG): container finished" podID="5bfa618f-330c-4aa6-9b3a-9aac777c1cf1" containerID="a8ce98336841503c7e8304748eae5f9baab8a2e64c0ad579193f94c4853c557a" exitCode=1
Jul 23 17:59:44.854801 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"c7c584cefa48fa5cea05944cdf25ffc0e38fcccbbf45f38540589d596139a340"}
Jul 23 17:59:44.854863 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"1e170f13dc7bc832cf494bdbc47666e1bf06af3a41fe712a751c7f9e0c0b05ea"}
Jul 23 17:59:44.854863 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"b9f57a745303dea43cf2bcc21640cec79221c808e0a02e8dc6235d0a6696c294"}
Jul 23 17:59:44.854966 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerDied","Data":"a8ce98336841503c7e8304748eae5f9baab8a2e64c0ad579193f94c4853c557a"}
Jul 23 17:59:44.854966 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.854917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"3d13164f0bf089f9b0f26849d78072db6e7a95acdbb78d09b00be2d975648393"}
Jul 23 17:59:44.856061 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.856038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lwdc5" event={"ID":"da29655f-7bd7-490f-926b-c40c5353dffc","Type":"ContainerStarted","Data":"1fc0a7809482848fdfb656cdf54f26afe13cf8555251048083cd6ea4dd4cc42f"}
Jul 23 17:59:44.857300 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.857279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9xbxt" event={"ID":"94942e6d-c9a9-4fb5-bac0-4aa81845e979","Type":"ContainerStarted","Data":"6b5c22240195657f57dba0e000a67f157f25c4dc074b368760e50eb2dec5ad6a"}
Jul 23 17:59:44.857412 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.857400    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:44.857816 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.857801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c789p"
Jul 23 17:59:44.892140 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.892095    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lxftc" podStartSLOduration=4.028600537 podStartE2EDuration="22.892079553s" totalImagesPullingTime="18.863479016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:44.89208066 +0000 UTC m=+23.682237775" watchObservedRunningTime="2026-07-23 17:59:44.892079553 +0000 UTC m=+23.682236666"
Jul 23 17:59:44.908456 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.908412    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9xbxt" podStartSLOduration=4.056785811 podStartE2EDuration="22.908398439s" totalImagesPullingTime="18.851612628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:44.90800791 +0000 UTC m=+23.698165024" watchObservedRunningTime="2026-07-23 17:59:44.908398439 +0000 UTC m=+23.698555554"
Jul 23 17:59:44.945326 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.945277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xjkjp" podStartSLOduration=4.102006322 podStartE2EDuration="22.945262381s" totalImagesPullingTime="18.843256059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:44.94493609 +0000 UTC m=+23.735093205" watchObservedRunningTime="2026-07-23 17:59:44.945262381 +0000 UTC m=+23.735419494"
Jul 23 17:59:44.961734 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:44.961690    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lwdc5" podStartSLOduration=4.12042085 podStartE2EDuration="22.961675802s" totalImagesPullingTime="18.841254952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:44.961561154 +0000 UTC m=+23.751718269" watchObservedRunningTime="2026-07-23 17:59:44.961675802 +0000 UTC m=+23.751832917"
Jul 23 17:59:45.049594 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.049569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:45.093159 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.093135    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 17:59:45.125192 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.125014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:45.571412 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.571219    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T17:59:45.09315386Z","UUID":"887b3dbb-13ab-4cd5-b8d4-600c0e6f14cc","Handler":null,"Name":"","Endpoint":""}
Jul 23 17:59:45.573088 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.573039    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 17:59:45.573088 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.573078    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 17:59:45.602166 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.602136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:45.602326 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.602171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:45.602326 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:45.602252    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:45.602415 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:45.602385    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:45.862430 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.862343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9" event={"ID":"d3bf21c2-367d-4495-9e6e-f4debd467e6a","Type":"ContainerStarted","Data":"d6f87fba28220a109b7ea455e10036337c70e89e14f6ca02309f71ce7e08d2ea"}
Jul 23 17:59:45.864569 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.864524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qfjkw" event={"ID":"4ea73276-b4e6-46cc-bf41-acb16a94142a","Type":"ContainerStarted","Data":"e1a49dee632c64a5a36f32342d77b6fac8f642df1dc33db1668007e000be532c"}
Jul 23 17:59:45.868212 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.868188    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 17:59:45.869014 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.868988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"6ae7398b8b61a3a5c37376d3fb6525b6b273cdf6f8e6a956a1f384ea56908c88"}
Jul 23 17:59:45.880410 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:45.880368    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qfjkw" podStartSLOduration=5.073731991 podStartE2EDuration="23.880348799s" totalImagesPullingTime="18.806616808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:45.87987618 +0000 UTC m=+24.670033295" watchObservedRunningTime="2026-07-23 17:59:45.880348799 +0000 UTC m=+24.670505914"
Jul 23 17:59:46.002036 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:46.001987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:46.103271 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:46.103244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:46.203709 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:46.203680    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:46.602363 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:46.602289    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:46.602597 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:46.602440    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:46.874380 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:46.874283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9" event={"ID":"d3bf21c2-367d-4495-9e6e-f4debd467e6a","Type":"ContainerStarted","Data":"a1e7ab9f7c40c336ac03b7a1448ec60ada8fb3f9faae5bd6db63947a5f0a6cbb"}
Jul 23 17:59:46.896179 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:46.896130    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfl9" podStartSLOduration=3.85734836 podStartE2EDuration="24.896115558s" totalImagesPullingTime="21.038767198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:46.895969987 +0000 UTC m=+25.686127102" watchObservedRunningTime="2026-07-23 17:59:46.896115558 +0000 UTC m=+25.686272672"
Jul 23 17:59:47.602867 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:47.602689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:47.603040 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:47.602742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:47.603040 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:47.602962    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:47.603108 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:47.603044    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:47.879891 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:47.879810    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 17:59:47.880334 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:47.880222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"46e5e8dc0b5bb3b071e462941ae2c52fd0f62d59e15474e028c0310b117f75c7"}
Jul 23 17:59:48.602049 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:48.602012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:48.602211 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:48.602152    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:49.602915 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.602672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:49.603519 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.602712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:49.603519 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:49.603076    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:49.603519 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:49.603199    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:49.886481 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.886384    2562 generic.go:356] "Generic (PLEG): container finished" podID="067b5e5d-0bd1-41cd-b076-7118f4472717" containerID="13fdf048df79718714908060aa1f8f5f06a805164bbfd2764290a0050c738b87" exitCode=0
Jul 23 17:59:49.886651 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.886493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerDied","Data":"13fdf048df79718714908060aa1f8f5f06a805164bbfd2764290a0050c738b87"}
Jul 23 17:59:49.890125 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.890105    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 17:59:49.890481 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.890460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"0c5c381fd2176baeb62bc259d5f42138d4a8155eea11da3d652247f4430bc817"}
Jul 23 17:59:49.890714 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.890690    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:49.890796 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.890751    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:49.890900 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.890874    2562 scope.go:117] "RemoveContainer" containerID="a8ce98336841503c7e8304748eae5f9baab8a2e64c0ad579193f94c4853c557a"
Jul 23 17:59:49.909304 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:49.909278    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:50.602009 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:50.601975    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:50.602276 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:50.602108    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:50.898164 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:50.898089    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 17:59:50.898611 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:50.898536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" event={"ID":"5bfa618f-330c-4aa6-9b3a-9aac777c1cf1","Type":"ContainerStarted","Data":"417e600acf5227e893142e5a69a630913d1b9fe32cf7dacffaafa39eabc1041e"}
Jul 23 17:59:50.898807 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:50.898738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:50.917206 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:50.917182    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 17:59:50.928002 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:50.927935    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq" podStartSLOduration=9.815762637 podStartE2EDuration="28.927920856s" totalImagesPullingTime="19.112158219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:50.927007476 +0000 UTC m=+29.717164590" watchObservedRunningTime="2026-07-23 17:59:50.927920856 +0000 UTC m=+29.718077970"
Jul 23 17:59:51.246048 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.245973    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2zwn"]
Jul 23 17:59:51.246280 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.246104    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:51.246280 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:51.246195    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:51.252323 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.252174    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vd8gk"]
Jul 23 17:59:51.252418 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.252400    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:51.252497 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:51.252481    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:51.254362 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.254337    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gdqpw"]
Jul 23 17:59:51.254466 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.254425    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:51.254539 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:51.254521    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:51.902659 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.902627    2562 generic.go:356] "Generic (PLEG): container finished" podID="067b5e5d-0bd1-41cd-b076-7118f4472717" containerID="6c2dd597cff2de7fe01781b7f6394947a1154535fff899e123a4c3abfb0c4c70" exitCode=0
Jul 23 17:59:51.903047 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:51.902721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerDied","Data":"6c2dd597cff2de7fe01781b7f6394947a1154535fff899e123a4c3abfb0c4c70"}
Jul 23 17:59:52.602223 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:52.602183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:52.602444 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:52.602245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:52.602444 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:52.602325    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:52.602444 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:52.602417    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:53.602454 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:53.602368    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:53.603053 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:53.602495    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:53.653425 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:53.653393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:53.749802 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:53.749769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:53.910434 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:53.910408    2562 generic.go:356] "Generic (PLEG): container finished" podID="067b5e5d-0bd1-41cd-b076-7118f4472717" containerID="75baf11d0add91841746f711a9bf547ae5a2b97952dc12b0451f70d61637e402" exitCode=0
Jul 23 17:59:53.910570 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:53.910452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerDied","Data":"75baf11d0add91841746f711a9bf547ae5a2b97952dc12b0451f70d61637e402"}
Jul 23 17:59:54.602093 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:54.602062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:54.602093 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:54.602091    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:54.602328 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:54.602171    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:54.602328 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:54.602246    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:55.373576 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:55.373545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:55.374295 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:55.373611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:55.374295 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.373698    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:55.374295 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.373721    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:55.374295 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.373762    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.373744196 +0000 UTC m=+66.163901292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:59:55.374295 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.373777    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret podName:fcbf1c01-1735-4764-9986-a750528b484e nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.373770461 +0000 UTC m=+66.163927554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret") pod "global-pull-secret-syncer-gdqpw" (UID: "fcbf1c01-1735-4764-9986-a750528b484e") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:59:55.474785 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:55.474753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:55.474994 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.474888    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:59:55.474994 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.474905    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:59:55.474994 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.474914    2562 projected.go:196] Error preparing data for projected volume kube-api-access-swbqd for pod openshift-network-diagnostics/network-check-target-z2zwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:55.474994 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.474994    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd podName:c5224216-d0f1-46ce-a6ef-d6f3ce2d640d nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.474975689 +0000 UTC m=+66.265132782 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-swbqd" (UniqueName: "kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd") pod "network-check-target-z2zwn" (UID: "c5224216-d0f1-46ce-a6ef-d6f3ce2d640d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:59:55.602467 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:55.602427    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:55.602667 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:55.602579    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:56.602425 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:56.602392    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:56.602826 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:56.602521    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2zwn" podUID="c5224216-d0f1-46ce-a6ef-d6f3ce2d640d"
Jul 23 17:59:56.602873 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:56.602393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:56.602969 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:56.602927    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gdqpw" podUID="fcbf1c01-1735-4764-9986-a750528b484e"
Jul 23 17:59:57.602372 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.602287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:57.602532 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:57.602414    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vd8gk" podUID="197e0944-75f2-4c47-8ed0-53194766062d"
Jul 23 17:59:57.603776 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.603751    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-3.ec2.internal" event="NodeReady"
Jul 23 17:59:57.603919 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.603906    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 17:59:57.662083 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.662045    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-77vs7"]
Jul 23 17:59:57.666827 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.666802    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.667994 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.667969    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vhlzz"]
Jul 23 17:59:57.671454 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.671434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.671999 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.671979    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lb24m\""
Jul 23 17:59:57.672102 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.672035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 17:59:57.672285 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.672267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 17:59:57.673920 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.673891    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 17:59:57.674117 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.673897    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 17:59:57.674189 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.674175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k2gdw\""
Jul 23 17:59:57.674243 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.674197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 17:59:57.677615 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.677597    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-77vs7"]
Jul 23 17:59:57.682192 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.682168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhlzz"]
Jul 23 17:59:57.768388 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.768357    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:57.792477 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.792440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd24080b-11ac-4d1a-85c5-f3928b81d06e-cert\") pod \"ingress-canary-vhlzz\" (UID: \"cd24080b-11ac-4d1a-85c5-f3928b81d06e\") " pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.792664 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.792488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.792664 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.792572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecda0a51-585e-43b0-89e0-b5c4e0a52004-tmp-dir\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.792664 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.792600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5jl\" (UniqueName: \"kubernetes.io/projected/ecda0a51-585e-43b0-89e0-b5c4e0a52004-kube-api-access-6k5jl\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.792664 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.792628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26ht\" (UniqueName: \"kubernetes.io/projected/cd24080b-11ac-4d1a-85c5-f3928b81d06e-kube-api-access-b26ht\") pod \"ingress-canary-vhlzz\" (UID: \"cd24080b-11ac-4d1a-85c5-f3928b81d06e\") " pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.792664 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.792655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecda0a51-585e-43b0-89e0-b5c4e0a52004-config-volume\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.866521 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.866433    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:57.893402 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.893368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd24080b-11ac-4d1a-85c5-f3928b81d06e-cert\") pod \"ingress-canary-vhlzz\" (UID: \"cd24080b-11ac-4d1a-85c5-f3928b81d06e\") " pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.893591 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.893413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.893591 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.893482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecda0a51-585e-43b0-89e0-b5c4e0a52004-tmp-dir\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.893591 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.893507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5jl\" (UniqueName: \"kubernetes.io/projected/ecda0a51-585e-43b0-89e0-b5c4e0a52004-kube-api-access-6k5jl\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.893591 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.893536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26ht\" (UniqueName: \"kubernetes.io/projected/cd24080b-11ac-4d1a-85c5-f3928b81d06e-kube-api-access-b26ht\") pod \"ingress-canary-vhlzz\" (UID: \"cd24080b-11ac-4d1a-85c5-f3928b81d06e\") " pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.893591 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.893559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecda0a51-585e-43b0-89e0-b5c4e0a52004-config-volume\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.893769 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:57.893627    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 17:59:57.893769 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:57.893697    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:58.393676349 +0000 UTC m=+37.183833455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 17:59:57.894082 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.894063    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecda0a51-585e-43b0-89e0-b5c4e0a52004-config-volume\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.894292 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.894272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecda0a51-585e-43b0-89e0-b5c4e0a52004-tmp-dir\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.898702 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.898671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd24080b-11ac-4d1a-85c5-f3928b81d06e-cert\") pod \"ingress-canary-vhlzz\" (UID: \"cd24080b-11ac-4d1a-85c5-f3928b81d06e\") " pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.903139 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.903114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5jl\" (UniqueName: \"kubernetes.io/projected/ecda0a51-585e-43b0-89e0-b5c4e0a52004-kube-api-access-6k5jl\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:57.903305 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.903288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26ht\" (UniqueName: \"kubernetes.io/projected/cd24080b-11ac-4d1a-85c5-f3928b81d06e-kube-api-access-b26ht\") pod \"ingress-canary-vhlzz\" (UID: \"cd24080b-11ac-4d1a-85c5-f3928b81d06e\") " pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:57.986123 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:57.985930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhlzz"
Jul 23 17:59:58.128564 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.128533    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhlzz"]
Jul 23 17:59:58.132400 ip-10-0-130-3 kubenswrapper[2562]: W0723 17:59:58.132374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd24080b_11ac_4d1a_85c5_f3928b81d06e.slice/crio-97135cd0ec5709ab8d93e05b829316e3e1eabe0740086b2b9af858c50eaf039b WatchSource:0}: Error finding container 97135cd0ec5709ab8d93e05b829316e3e1eabe0740086b2b9af858c50eaf039b: Status 404 returned error can't find the container with id 97135cd0ec5709ab8d93e05b829316e3e1eabe0740086b2b9af858c50eaf039b
Jul 23 17:59:58.396832 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.396739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:58.396996 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:58.396859    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 17:59:58.396996 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:58.396912    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:59.396898477 +0000 UTC m=+38.187055570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 17:59:58.602138 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.602106    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 17:59:58.602291 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.602166    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 17:59:58.605124 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.605097    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 17:59:58.605506 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.605127    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 17:59:58.605506 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.605159    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 17:59:58.605506 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.605360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbh2j\""
Jul 23 17:59:58.924878 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.924837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhlzz" event={"ID":"cd24080b-11ac-4d1a-85c5-f3928b81d06e","Type":"ContainerStarted","Data":"97135cd0ec5709ab8d93e05b829316e3e1eabe0740086b2b9af858c50eaf039b"}
Jul 23 17:59:58.928081 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.928054    2562 generic.go:356] "Generic (PLEG): container finished" podID="067b5e5d-0bd1-41cd-b076-7118f4472717" containerID="0feca9b8066e846a3dd90000af6990f876174a9bc5d255454d42577ecef88c4e" exitCode=0
Jul 23 17:59:58.928203 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:58.928090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerDied","Data":"0feca9b8066e846a3dd90000af6990f876174a9bc5d255454d42577ecef88c4e"}
Jul 23 17:59:59.404103 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.404070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 17:59:59.404263 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:59.404188    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 17:59:59.404263 ip-10-0-130-3 kubenswrapper[2562]: E0723 17:59:59.404242    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:01.404228088 +0000 UTC m=+40.194385181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 17:59:59.498299 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.498270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:59.601829 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.601745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log" limit=1024
Jul 23 17:59:59.602869 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.602103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 17:59:59.604818 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.604788    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 17:59:59.604818 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.604811    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hv4kc\""
Jul 23 17:59:59.934783 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.934551    2562 generic.go:356] "Generic (PLEG): container finished" podID="067b5e5d-0bd1-41cd-b076-7118f4472717" containerID="e1ae33a94ae7ca006cefe63ffc64c59390a4a01f30974c512c5488f581e4e0f5" exitCode=0
Jul 23 17:59:59.935238 ip-10-0-130-3 kubenswrapper[2562]: I0723 17:59:59.934638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerDied","Data":"e1ae33a94ae7ca006cefe63ffc64c59390a4a01f30974c512c5488f581e4e0f5"}
Jul 23 18:00:00.941366 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:00.941281    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" event={"ID":"067b5e5d-0bd1-41cd-b076-7118f4472717","Type":"ContainerStarted","Data":"6c8aef9750023e5b1c26230e25dfc696e90eede071f93ac76e3977e3ecc7dd5a"}
Jul 23 18:00:00.942752 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:00.942731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhlzz" event={"ID":"cd24080b-11ac-4d1a-85c5-f3928b81d06e","Type":"ContainerStarted","Data":"b5a62760af582fe38ee970438b4cb7e3d9c550675af7a75f30acac1d80bfa66b"}
Jul 23 18:00:00.973380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:00.973313    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lqf6q" podStartSLOduration=8.982931884 podStartE2EDuration="38.973295148s" totalImagesPullingTime="29.990363264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:00.972157777 +0000 UTC m=+39.762314940" watchObservedRunningTime="2026-07-23 18:00:00.973295148 +0000 UTC m=+39.763452263"
Jul 23 18:00:01.420140 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:01.420092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 18:00:01.420314 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:01.420229    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 18:00:01.420314 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:01.420295    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:05.420279069 +0000 UTC m=+44.210436176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 18:00:05.447775 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:05.447736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 18:00:05.448172 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:05.447905    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 18:00:05.448172 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:05.447986    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:13.447970614 +0000 UTC m=+52.238127707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 18:00:11.691622 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.691560    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vhlzz" podStartSLOduration=12.139896101 podStartE2EDuration="14.691543886s" totalImagesPullingTime="2.551647785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:00.997661365 +0000 UTC m=+39.787818491" watchObservedRunningTime="2026-07-23 18:00:11.691543886 +0000 UTC m=+50.481701042"
Jul 23 18:00:11.692336 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.692317    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"]
Jul 23 18:00:11.710431 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.710392    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f7cc9cdfd-b5f4s"]
Jul 23 18:00:11.710571 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.710545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"
Jul 23 18:00:11.713286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.713267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lwmzb\""
Jul 23 18:00:11.713401 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.713296    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 23 18:00:11.713401 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.713268    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 23 18:00:11.722776 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.722757    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-f72dk"]
Jul 23 18:00:11.722907 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.722890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.725330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.725311    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 23 18:00:11.725432 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.725316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 23 18:00:11.725432 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.725357    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 23 18:00:11.725675 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.725659    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 23 18:00:11.725907 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.725892    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 18:00:11.725977 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.725927    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 18:00:11.727200 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.727184    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k5h4q\""
Jul 23 18:00:11.738974 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.738953    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"]
Jul 23 18:00:11.739067 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.738994    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f7cc9cdfd-b5f4s"]
Jul 23 18:00:11.739067 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.739010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-f72dk"]
Jul 23 18:00:11.754611 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.754593    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.756899 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.756882    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 23 18:00:11.757014 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.756985    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 23 18:00:11.757065 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.757044    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 18:00:11.757187 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.757174    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6fm2g\""
Jul 23 18:00:11.757313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.757299    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 18:00:11.762593 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.762576    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 23 18:00:11.889992 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.889960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1abe68d-f131-43c3-86b5-224632d4bd30-tmp\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.889992 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.889996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-stats-auth\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1abe68d-f131-43c3-86b5-224632d4bd30-service-ca-bundle\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1abe68d-f131-43c3-86b5-224632d4bd30-serving-cert\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-default-certificate\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxszt\" (UniqueName: \"kubernetes.io/projected/8bb7fffa-642d-4201-ba63-f62a685a566c-kube-api-access-dxszt\") pod \"volume-data-source-validator-78cf4c6f6-znpxf\" (UID: \"8bb7fffa-642d-4201-ba63-f62a685a566c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1abe68d-f131-43c3-86b5-224632d4bd30-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.890229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4226k\" (UniqueName: \"kubernetes.io/projected/aab248f1-1f0e-4550-abfb-ccf8deff11e6-kube-api-access-4226k\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.890437 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1abe68d-f131-43c3-86b5-224632d4bd30-snapshots\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.890437 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.890437 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.890297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjfz\" (UniqueName: \"kubernetes.io/projected/d1abe68d-f131-43c3-86b5-224632d4bd30-kube-api-access-vrjfz\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991295 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1abe68d-f131-43c3-86b5-224632d4bd30-tmp\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991295 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-stats-auth\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.991295 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:11.991345    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:11.991402    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:12.491386931 +0000 UTC m=+51.281544028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : secret "router-metrics-certs-default" not found
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1abe68d-f131-43c3-86b5-224632d4bd30-service-ca-bundle\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1abe68d-f131-43c3-86b5-224632d4bd30-serving-cert\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-default-certificate\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxszt\" (UniqueName: \"kubernetes.io/projected/8bb7fffa-642d-4201-ba63-f62a685a566c-kube-api-access-dxszt\") pod \"volume-data-source-validator-78cf4c6f6-znpxf\" (UID: \"8bb7fffa-642d-4201-ba63-f62a685a566c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1abe68d-f131-43c3-86b5-224632d4bd30-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4226k\" (UniqueName: \"kubernetes.io/projected/aab248f1-1f0e-4550-abfb-ccf8deff11e6-kube-api-access-4226k\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1abe68d-f131-43c3-86b5-224632d4bd30-snapshots\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1abe68d-f131-43c3-86b5-224632d4bd30-tmp\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.991729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.991677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjfz\" (UniqueName: \"kubernetes.io/projected/d1abe68d-f131-43c3-86b5-224632d4bd30-kube-api-access-vrjfz\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.992275 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.992014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1abe68d-f131-43c3-86b5-224632d4bd30-service-ca-bundle\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.992275 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:11.992122    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:12.492103713 +0000 UTC m=+51.282260820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : configmap references non-existent config key: service-ca.crt
Jul 23 18:00:11.992275 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.992187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1abe68d-f131-43c3-86b5-224632d4bd30-snapshots\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.992696 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.992674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1abe68d-f131-43c3-86b5-224632d4bd30-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.995398 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.995368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1abe68d-f131-43c3-86b5-224632d4bd30-serving-cert\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:11.995520 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.995482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-stats-auth\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.995560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.995525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-default-certificate\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:11.999736 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.999710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxszt\" (UniqueName: \"kubernetes.io/projected/8bb7fffa-642d-4201-ba63-f62a685a566c-kube-api-access-dxszt\") pod \"volume-data-source-validator-78cf4c6f6-znpxf\" (UID: \"8bb7fffa-642d-4201-ba63-f62a685a566c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"
Jul 23 18:00:11.999837 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:11.999779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4226k\" (UniqueName: \"kubernetes.io/projected/aab248f1-1f0e-4550-abfb-ccf8deff11e6-kube-api-access-4226k\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:12.000207 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.000187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjfz\" (UniqueName: \"kubernetes.io/projected/d1abe68d-f131-43c3-86b5-224632d4bd30-kube-api-access-vrjfz\") pod \"insights-operator-555d9d7885-f72dk\" (UID: \"d1abe68d-f131-43c3-86b5-224632d4bd30\") " pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:12.020651 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.020632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"
Jul 23 18:00:12.062476 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.062436    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-f72dk"
Jul 23 18:00:12.189800 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.189769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf"]
Jul 23 18:00:12.193127 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:12.193092    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb7fffa_642d_4201_ba63_f62a685a566c.slice/crio-0e24cb494f795a4150e475492f5a89a14625b4ab37408f8c1c9a94c30345dacc WatchSource:0}: Error finding container 0e24cb494f795a4150e475492f5a89a14625b4ab37408f8c1c9a94c30345dacc: Status 404 returned error can't find the container with id 0e24cb494f795a4150e475492f5a89a14625b4ab37408f8c1c9a94c30345dacc
Jul 23 18:00:12.209203 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.209180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-f72dk"]
Jul 23 18:00:12.212711 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:12.212682    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1abe68d_f131_43c3_86b5_224632d4bd30.slice/crio-ddfe4753f2aaa3de4f65c6eb689f1801fb2b8b8a463457518e7502b274708b87 WatchSource:0}: Error finding container ddfe4753f2aaa3de4f65c6eb689f1801fb2b8b8a463457518e7502b274708b87: Status 404 returned error can't find the container with id ddfe4753f2aaa3de4f65c6eb689f1801fb2b8b8a463457518e7502b274708b87
Jul 23 18:00:12.495431 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.495391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:12.495646 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.495477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:12.495646 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:12.495581    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 18:00:12.495646 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:12.495587    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:13.495565134 +0000 UTC m=+52.285722227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : configmap references non-existent config key: service-ca.crt
Jul 23 18:00:12.495646 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:12.495637    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:13.495621224 +0000 UTC m=+52.285778319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : secret "router-metrics-certs-default" not found
Jul 23 18:00:12.971317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.971278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-f72dk" event={"ID":"d1abe68d-f131-43c3-86b5-224632d4bd30","Type":"ContainerStarted","Data":"ddfe4753f2aaa3de4f65c6eb689f1801fb2b8b8a463457518e7502b274708b87"}
Jul 23 18:00:12.972817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:12.972776    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf" event={"ID":"8bb7fffa-642d-4201-ba63-f62a685a566c","Type":"ContainerStarted","Data":"0e24cb494f795a4150e475492f5a89a14625b4ab37408f8c1c9a94c30345dacc"}
Jul 23 18:00:13.503723 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:13.503683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:13.503887 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:13.503741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 18:00:13.503887 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:13.503767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:13.503887 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:13.503852    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 18:00:13.504020 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:13.503902    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:15.503884247 +0000 UTC m=+54.294041357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : configmap references non-existent config key: service-ca.crt
Jul 23 18:00:13.504020 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:13.503911    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 18:00:13.504020 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:13.503923    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:15.503913955 +0000 UTC m=+54.294071051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : secret "router-metrics-certs-default" not found
Jul 23 18:00:13.504020 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:13.503968    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:29.503938245 +0000 UTC m=+68.294095341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 18:00:14.980502 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:14.980403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf" event={"ID":"8bb7fffa-642d-4201-ba63-f62a685a566c","Type":"ContainerStarted","Data":"9ab4ba57929f180bd37412b03669d42699c8344fa8ef76503ce2ea98326ca5f6"}
Jul 23 18:00:14.982199 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:14.982170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-f72dk" event={"ID":"d1abe68d-f131-43c3-86b5-224632d4bd30","Type":"ContainerStarted","Data":"6afd93a59a0c1ac1933466cf427c7b44a49f726862e368c9ec7c2a30058e06b1"}
Jul 23 18:00:14.995872 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:14.995819    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-znpxf" podStartSLOduration=1.848261645 podStartE2EDuration="3.995804218s" totalImagesPullingTime="2.147542573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:14.99501493 +0000 UTC m=+53.785172047" watchObservedRunningTime="2026-07-23 18:00:14.995804218 +0000 UTC m=+53.785961333"
Jul 23 18:00:15.012399 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:15.012354    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-f72dk" podStartSLOduration=1.5312404069999999 podStartE2EDuration="4.012339367s" totalImagesPullingTime="2.48109896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:15.011306526 +0000 UTC m=+53.801463642" watchObservedRunningTime="2026-07-23 18:00:15.012339367 +0000 UTC m=+53.802496482"
Jul 23 18:00:15.518991 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:15.518961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:15.519159 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:15.519025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:15.519159 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:15.519123    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 18:00:15.519239 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:15.519164    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:19.519149092 +0000 UTC m=+58.309306188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : configmap references non-existent config key: service-ca.crt
Jul 23 18:00:15.519239 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:15.519185    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:19.519172475 +0000 UTC m=+58.309329569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : secret "router-metrics-certs-default" not found
Jul 23 18:00:16.569198 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.569167    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nlnq4"]
Jul 23 18:00:16.572312 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.572288    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.574798 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.574775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 18:00:16.574914 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.574815    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 18:00:16.574914 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.574845    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mq246\""
Jul 23 18:00:16.585723 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.585700    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nlnq4"]
Jul 23 18:00:16.629761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.629731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-crio-socket\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.629761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.629765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-data-volume\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.629983 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.629788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.629983 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.629810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.629983 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.629844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdn4\" (UniqueName: \"kubernetes.io/projected/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-api-access-ptdn4\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730571 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-crio-socket\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730571 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-data-volume\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdn4\" (UniqueName: \"kubernetes.io/projected/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-api-access-ptdn4\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-crio-socket\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:16.730696    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:16.730731    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:16.730759    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:17.230746623 +0000 UTC m=+56.020903717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : configmap "kube-rbac-proxy" not found
Jul 23 18:00:16.730806 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:16.730783    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:17.230767731 +0000 UTC m=+56.020924849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:16.731116 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.730863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-data-volume\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:16.739331 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:16.739308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdn4\" (UniqueName: \"kubernetes.io/projected/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-api-access-ptdn4\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:17.233210 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:17.233172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:17.233210 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:17.233215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:17.233418 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:17.233311    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 18:00:17.233418 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:17.233343    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:17.233418 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:17.233380    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:18.233364259 +0000 UTC m=+57.023521356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : configmap "kube-rbac-proxy" not found
Jul 23 18:00:17.233418 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:17.233394    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:18.233388478 +0000 UTC m=+57.023545571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:18.239344 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:18.239309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:18.239344 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:18.239347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:18.239808 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:18.239444    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 18:00:18.239808 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:18.239454    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:18.239808 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:18.239513    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:20.239499527 +0000 UTC m=+59.029656620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:18.239808 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:18.239526    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:20.239519452 +0000 UTC m=+59.029676545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : configmap "kube-rbac-proxy" not found
Jul 23 18:00:18.314425 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:18.314401    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9xbxt_94942e6d-c9a9-4fb5-bac0-4aa81845e979/dns-node-resolver/0.log"
Jul 23 18:00:19.514828 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:19.514801    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lwdc5_da29655f-7bd7-490f-926b-c40c5353dffc/node-ca/0.log"
Jul 23 18:00:19.546281 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:19.546235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:19.546444 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:19.546315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:19.546444 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:19.546370    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 18:00:19.546444 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:19.546439    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.546421282 +0000 UTC m=+66.336578381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : secret "router-metrics-certs-default" not found
Jul 23 18:00:19.546604 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:19.546496    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:27.54647648 +0000 UTC m=+66.336633593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : configmap references non-existent config key: service-ca.crt
Jul 23 18:00:20.250958 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:20.250917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:20.251135 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:20.251024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:20.251135 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:20.251035    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 18:00:20.251135 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:20.251101    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:24.251081066 +0000 UTC m=+63.041238181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : configmap "kube-rbac-proxy" not found
Jul 23 18:00:20.251245 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:20.251183    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:20.251276 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:20.251246    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:24.251231417 +0000 UTC m=+63.041388510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:20.315673 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:20.315646    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhlzz_cd24080b-11ac-4d1a-85c5-f3928b81d06e/serve-healthcheck-canary/0.log"
Jul 23 18:00:22.822435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.822399    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"]
Jul 23 18:00:22.850340 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.850314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"]
Jul 23 18:00:22.850497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.850433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"
Jul 23 18:00:22.852619 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.852594    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 23 18:00:22.852745 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.852640    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8xwjp\""
Jul 23 18:00:22.853329 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.853314    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 23 18:00:22.867163 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.867143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flf9h\" (UniqueName: \"kubernetes.io/projected/a66476a4-ff22-450d-a5d2-d96333402bc6-kube-api-access-flf9h\") pod \"migrator-7cc9c7ffd7-nd6rt\" (UID: \"a66476a4-ff22-450d-a5d2-d96333402bc6\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"
Jul 23 18:00:22.927070 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.927041    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbfsq"
Jul 23 18:00:22.968124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.968090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flf9h\" (UniqueName: \"kubernetes.io/projected/a66476a4-ff22-450d-a5d2-d96333402bc6-kube-api-access-flf9h\") pod \"migrator-7cc9c7ffd7-nd6rt\" (UID: \"a66476a4-ff22-450d-a5d2-d96333402bc6\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"
Jul 23 18:00:22.981358 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:22.981328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flf9h\" (UniqueName: \"kubernetes.io/projected/a66476a4-ff22-450d-a5d2-d96333402bc6-kube-api-access-flf9h\") pod \"migrator-7cc9c7ffd7-nd6rt\" (UID: \"a66476a4-ff22-450d-a5d2-d96333402bc6\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"
Jul 23 18:00:23.159541 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:23.159461    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"
Jul 23 18:00:23.284026 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:23.283995    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt"]
Jul 23 18:00:23.287285 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:23.287252    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66476a4_ff22_450d_a5d2_d96333402bc6.slice/crio-1a4de23d8e36c51a661d78cd1600644d782828285f35809737f635f6cdd9a26a WatchSource:0}: Error finding container 1a4de23d8e36c51a661d78cd1600644d782828285f35809737f635f6cdd9a26a: Status 404 returned error can't find the container with id 1a4de23d8e36c51a661d78cd1600644d782828285f35809737f635f6cdd9a26a
Jul 23 18:00:24.008833 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:24.008799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt" event={"ID":"a66476a4-ff22-450d-a5d2-d96333402bc6","Type":"ContainerStarted","Data":"1a4de23d8e36c51a661d78cd1600644d782828285f35809737f635f6cdd9a26a"}
Jul 23 18:00:24.278919 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:24.278833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:24.278919 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:24.278876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:24.279166 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:24.278972    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 18:00:24.279166 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:24.279017    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:24.279166 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:24.279041    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:32.279022491 +0000 UTC m=+71.069179585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : configmap "kube-rbac-proxy" not found
Jul 23 18:00:24.279166 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:24.279058    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:32.279050043 +0000 UTC m=+71.069207138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:25.013560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:25.013524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt" event={"ID":"a66476a4-ff22-450d-a5d2-d96333402bc6","Type":"ContainerStarted","Data":"dbe6f3aaf4e9655dfa194761584e42102a365461c118fc9c9cc8322c81dbbe7f"}
Jul 23 18:00:25.014037 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:25.013622    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt" event={"ID":"a66476a4-ff22-450d-a5d2-d96333402bc6","Type":"ContainerStarted","Data":"9c88d2ae8585307b740b4bf941b0e145f9b76814195b4dfac34b172a3664dddc"}
Jul 23 18:00:27.400907 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.400869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 18:00:27.401341 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.400922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 18:00:27.403366 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.403346    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 18:00:27.403421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.403351    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 18:00:27.411712 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:27.411696    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 18:00:27.411793 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:27.411780    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs podName:197e0944-75f2-4c47-8ed0-53194766062d nodeName:}" failed. No retries permitted until 2026-07-23 18:01:31.411763596 +0000 UTC m=+130.201920689 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs") pod "network-metrics-daemon-vd8gk" (UID: "197e0944-75f2-4c47-8ed0-53194766062d") : secret "metrics-daemon-secret" not found
Jul 23 18:00:27.413924 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.413901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fcbf1c01-1735-4764-9986-a750528b484e-original-pull-secret\") pod \"global-pull-secret-syncer-gdqpw\" (UID: \"fcbf1c01-1735-4764-9986-a750528b484e\") " pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 18:00:27.419882 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.419861    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gdqpw"
Jul 23 18:00:27.501421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.501368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 18:00:27.504052 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.504032    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 18:00:27.514875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.514852    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 18:00:27.525348 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.525313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbqd\" (UniqueName: \"kubernetes.io/projected/c5224216-d0f1-46ce-a6ef-d6f3ce2d640d-kube-api-access-swbqd\") pod \"network-check-target-z2zwn\" (UID: \"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d\") " pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 18:00:27.546502 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.546442    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-nd6rt" podStartSLOduration=4.385976649 podStartE2EDuration="5.546427332s" totalImagesPullingTime="1.160450683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:25.040272528 +0000 UTC m=+63.830429642" watchObservedRunningTime="2026-07-23 18:00:27.546427332 +0000 UTC m=+66.336584466"
Jul 23 18:00:27.547286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.547259    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gdqpw"]
Jul 23 18:00:27.550296 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:27.550275    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbf1c01_1735_4764_9986_a750528b484e.slice/crio-3c50e436d51d5bcd3a2ff2a08767c89ac69e08a8fe87d9d87b824e445a20c23d WatchSource:0}: Error finding container 3c50e436d51d5bcd3a2ff2a08767c89ac69e08a8fe87d9d87b824e445a20c23d: Status 404 returned error can't find the container with id 3c50e436d51d5bcd3a2ff2a08767c89ac69e08a8fe87d9d87b824e445a20c23d
Jul 23 18:00:27.602652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.602617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:27.602787 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.602671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:27.602787 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:27.602770    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 18:00:27.602869 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:27.602815    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:43.602799683 +0000 UTC m=+82.392956784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : configmap references non-existent config key: service-ca.crt
Jul 23 18:00:27.602869 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:27.602830    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs podName:aab248f1-1f0e-4550-abfb-ccf8deff11e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:43.602823955 +0000 UTC m=+82.392981050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs") pod "router-default-f7cc9cdfd-b5f4s" (UID: "aab248f1-1f0e-4550-abfb-ccf8deff11e6") : secret "router-metrics-certs-default" not found
Jul 23 18:00:27.715623 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.715529    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbh2j\""
Jul 23 18:00:27.723480 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.723461    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 18:00:27.849398 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:27.849367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2zwn"]
Jul 23 18:00:27.852317 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:27.852290    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5224216_d0f1_46ce_a6ef_d6f3ce2d640d.slice/crio-10fa42807ae2c69ee68b858fe072edcbfe782515c8c9a3c332d05f97307779e0 WatchSource:0}: Error finding container 10fa42807ae2c69ee68b858fe072edcbfe782515c8c9a3c332d05f97307779e0: Status 404 returned error can't find the container with id 10fa42807ae2c69ee68b858fe072edcbfe782515c8c9a3c332d05f97307779e0
Jul 23 18:00:28.022693 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:28.022658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gdqpw" event={"ID":"fcbf1c01-1735-4764-9986-a750528b484e","Type":"ContainerStarted","Data":"3c50e436d51d5bcd3a2ff2a08767c89ac69e08a8fe87d9d87b824e445a20c23d"}
Jul 23 18:00:28.023780 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:28.023753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2zwn" event={"ID":"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d","Type":"ContainerStarted","Data":"10fa42807ae2c69ee68b858fe072edcbfe782515c8c9a3c332d05f97307779e0"}
Jul 23 18:00:29.518248 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:29.518206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 18:00:29.518644 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:29.518384    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 18:00:29.518644 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:29.518461    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls podName:ecda0a51-585e-43b0-89e0-b5c4e0a52004 nodeName:}" failed. No retries permitted until 2026-07-23 18:01:01.518440736 +0000 UTC m=+100.308597849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls") pod "dns-default-77vs7" (UID: "ecda0a51-585e-43b0-89e0-b5c4e0a52004") : secret "dns-default-metrics-tls" not found
Jul 23 18:00:32.346182 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:32.346145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:32.346653 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:32.346200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:32.346653 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:32.346375    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:32.346653 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:00:32.346453    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls podName:9bf9f4de-2e4d-4981-a100-9d1c91ba02f5 nodeName:}" failed. No retries permitted until 2026-07-23 18:00:48.346431103 +0000 UTC m=+87.136588196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlnq4" (UID: "9bf9f4de-2e4d-4981-a100-9d1c91ba02f5") : secret "insights-runtime-extractor-tls" not found
Jul 23 18:00:32.346809 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:32.346790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:33.043778 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:33.043745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gdqpw" event={"ID":"fcbf1c01-1735-4764-9986-a750528b484e","Type":"ContainerStarted","Data":"f2685aff507aa8eedc45b9b2416d7a22fcdb8ed9d48bfe78012ab049f4df3d44"}
Jul 23 18:00:33.045161 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:33.045135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2zwn" event={"ID":"c5224216-d0f1-46ce-a6ef-d6f3ce2d640d","Type":"ContainerStarted","Data":"8ea2f9ef32971c311ab6f50bf1b6f045b016c23f85a91e08f296130d80c59450"}
Jul 23 18:00:33.045284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:33.045273    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 18:00:33.074901 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:33.074842    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gdqpw" podStartSLOduration=66.033114009 podStartE2EDuration="1m11.07482235s" totalImagesPullingTime="5.041708341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:33.074085137 +0000 UTC m=+71.864242265" watchObservedRunningTime="2026-07-23 18:00:33.07482235 +0000 UTC m=+71.864979463"
Jul 23 18:00:33.094850 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:33.094808    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z2zwn" podStartSLOduration=66.359310181 podStartE2EDuration="1m11.094794131s" totalImagesPullingTime="4.73548395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:33.094226125 +0000 UTC m=+71.884383266" watchObservedRunningTime="2026-07-23 18:00:33.094794131 +0000 UTC m=+71.884951294"
Jul 23 18:00:43.634456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.634329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:43.634456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.634391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:43.635120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.635103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab248f1-1f0e-4550-abfb-ccf8deff11e6-service-ca-bundle\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:43.636856 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.636839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab248f1-1f0e-4550-abfb-ccf8deff11e6-metrics-certs\") pod \"router-default-f7cc9cdfd-b5f4s\" (UID: \"aab248f1-1f0e-4550-abfb-ccf8deff11e6\") " pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:43.833963 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.833905    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k5h4q\""
Jul 23 18:00:43.842262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.842233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:43.972593 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:43.972562    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f7cc9cdfd-b5f4s"]
Jul 23 18:00:43.975885 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:43.975852    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab248f1_1f0e_4550_abfb_ccf8deff11e6.slice/crio-71eceb72294f0adb6e5a011a585f937141516c07df58556ac927718c5151da6e WatchSource:0}: Error finding container 71eceb72294f0adb6e5a011a585f937141516c07df58556ac927718c5151da6e: Status 404 returned error can't find the container with id 71eceb72294f0adb6e5a011a585f937141516c07df58556ac927718c5151da6e
Jul 23 18:00:44.081150 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:44.081114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s" event={"ID":"aab248f1-1f0e-4550-abfb-ccf8deff11e6","Type":"ContainerStarted","Data":"8155ec7ecf588d5b6e64346dad8d817e647bd89d8c90a147222aabd792c5ef50"}
Jul 23 18:00:44.081309 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:44.081163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s" event={"ID":"aab248f1-1f0e-4550-abfb-ccf8deff11e6","Type":"ContainerStarted","Data":"71eceb72294f0adb6e5a011a585f937141516c07df58556ac927718c5151da6e"}
Jul 23 18:00:44.105846 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:44.105797    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s" podStartSLOduration=33.105758317 podStartE2EDuration="33.105758317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:44.105017343 +0000 UTC m=+82.895174482" watchObservedRunningTime="2026-07-23 18:00:44.105758317 +0000 UTC m=+82.895915469"
Jul 23 18:00:44.842742 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:44.842698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:44.845369 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:44.845346    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:45.083925 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:45.083895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:45.085135 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:45.085116    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f7cc9cdfd-b5f4s"
Jul 23 18:00:46.498909 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.498867    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"]
Jul 23 18:00:46.503140 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.503117    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.528346 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.528324    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 18:00:46.528490 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.528410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 18:00:46.529236 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.529220    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 18:00:46.529314 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.529233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 18:00:46.534989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.531909    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 18:00:46.534989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.532013    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 18:00:46.534989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.532091    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 18:00:46.537766 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.537740    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"]
Jul 23 18:00:46.557707 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.557676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/1639f7a5-f428-48ae-ba69-2dc68f200c09-kube-api-access-skfzh\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.557707 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.557710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.557916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.557765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639f7a5-f428-48ae-ba69-2dc68f200c09-additional-service-ca\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.557916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.557793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-hub\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.557916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.557813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-ca\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.558054 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.557921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.658436 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.658399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639f7a5-f428-48ae-ba69-2dc68f200c09-additional-service-ca\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.658436 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.658442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-hub\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.658692 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.658466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-ca\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.658692 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.658643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.658801 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.658698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/1639f7a5-f428-48ae-ba69-2dc68f200c09-kube-api-access-skfzh\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.658861 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.658837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.659538 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.659514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639f7a5-f428-48ae-ba69-2dc68f200c09-additional-service-ca\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.661229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.661207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-hub\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.661364 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.661336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.661603 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.661585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-ca\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.661753 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.661732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1639f7a5-f428-48ae-ba69-2dc68f200c09-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.683539 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.683508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfzh\" (UniqueName: \"kubernetes.io/projected/1639f7a5-f428-48ae-ba69-2dc68f200c09-kube-api-access-skfzh\") pod \"cluster-proxy-proxy-agent-769c579cc5-5pw64\" (UID: \"1639f7a5-f428-48ae-ba69-2dc68f200c09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.825222 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.825188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"
Jul 23 18:00:46.965884 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:46.965853    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64"]
Jul 23 18:00:46.968761 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:46.968739    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1639f7a5_f428_48ae_ba69_2dc68f200c09.slice/crio-cf3f8cbffca01d674c5238751913f9b97f7d1e89c28481d1ba71746765eeca0e WatchSource:0}: Error finding container cf3f8cbffca01d674c5238751913f9b97f7d1e89c28481d1ba71746765eeca0e: Status 404 returned error can't find the container with id cf3f8cbffca01d674c5238751913f9b97f7d1e89c28481d1ba71746765eeca0e
Jul 23 18:00:47.090505 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:47.090415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64" event={"ID":"1639f7a5-f428-48ae-ba69-2dc68f200c09","Type":"ContainerStarted","Data":"cf3f8cbffca01d674c5238751913f9b97f7d1e89c28481d1ba71746765eeca0e"}
Jul 23 18:00:48.371230 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:48.371192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:48.374078 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:48.374040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bf9f4de-2e4d-4981-a100-9d1c91ba02f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlnq4\" (UID: \"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5\") " pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:48.383057 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:48.383027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mq246\""
Jul 23 18:00:48.390977 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:48.390929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nlnq4"
Jul 23 18:00:48.552332 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:48.552296    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nlnq4"]
Jul 23 18:00:48.556882 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:00:48.556850    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf9f4de_2e4d_4981_a100_9d1c91ba02f5.slice/crio-4791d9b11093ae3b024ffa4c9a84ded7ba6aedeb7d10ccc812261a3201b1bf16 WatchSource:0}: Error finding container 4791d9b11093ae3b024ffa4c9a84ded7ba6aedeb7d10ccc812261a3201b1bf16: Status 404 returned error can't find the container with id 4791d9b11093ae3b024ffa4c9a84ded7ba6aedeb7d10ccc812261a3201b1bf16
Jul 23 18:00:49.098713 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:49.098666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlnq4" event={"ID":"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5","Type":"ContainerStarted","Data":"4791d9b11093ae3b024ffa4c9a84ded7ba6aedeb7d10ccc812261a3201b1bf16"}
Jul 23 18:00:52.114487 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:52.114451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64" event={"ID":"1639f7a5-f428-48ae-ba69-2dc68f200c09","Type":"ContainerStarted","Data":"e4431ee43a889296b2ac1c90f4ba6138e3d89b313663db7536c0ade5929a4b76"}
Jul 23 18:00:52.114891 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:52.114506    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64" event={"ID":"1639f7a5-f428-48ae-ba69-2dc68f200c09","Type":"ContainerStarted","Data":"7ba23c75e125e1cf3d0d33620448a0e21f1e78163a678829b81e19798ce44220"}
Jul 23 18:00:55.128005 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:55.127963    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64" event={"ID":"1639f7a5-f428-48ae-ba69-2dc68f200c09","Type":"ContainerStarted","Data":"390f7ebe96944badfbf019350193c53aa23f12a28f6064fc7ed0c0ffd35d2c96"}
Jul 23 18:00:55.129515 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:55.129492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlnq4" event={"ID":"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5","Type":"ContainerStarted","Data":"97409a6b454377318b2611fc348efa68269b0cf4262e4cd7157168933bab8e0d"}
Jul 23 18:00:55.151277 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:55.151224    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-769c579cc5-5pw64" podStartSLOduration=4.242800418 podStartE2EDuration="9.15120939s" totalImagesPullingTime="4.908408972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:00:55.149710861 +0000 UTC m=+93.939867989" watchObservedRunningTime="2026-07-23 18:00:55.15120939 +0000 UTC m=+93.941366560"
Jul 23 18:00:59.147132 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:00:59.147087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlnq4" event={"ID":"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5","Type":"ContainerStarted","Data":"c5c574a245d0fcdbb96df595849e35c53e0ad4b100436886b7d74de89ecf3803"}
Jul 23 18:01:00.372625 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.372594    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"]
Jul 23 18:01:00.375886 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.375864    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7mmpd"]
Jul 23 18:01:00.376043 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.376016    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.378877 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.378843    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.383683 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.383649    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 18:01:00.383838 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.383817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 23 18:01:00.384608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.384308    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 23 18:01:00.384608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.384326    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 18:01:00.384608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.384350    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-br47v\""
Jul 23 18:01:00.384608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.384402    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 23 18:01:00.384608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.384307    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 18:01:00.387813 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.387796    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 18:01:00.388089 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.388075    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 18:01:00.388280 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.388267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 18:01:00.389083 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.389065    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5ghbr\""
Jul 23 18:01:00.403756 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.403733    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"]
Jul 23 18:01:00.475873 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.475823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.475873 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.475867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-textfile\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.475913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9927a-f08f-42f7-8668-278ec11cd4d5-metrics-client-ca\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.475934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.475985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f54a589b-695d-45df-b90d-52d4581118d4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.476110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-tls\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54a589b-695d-45df-b90d-52d4581118d4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.476110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-wtmp\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-root\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkd4\" (UniqueName: \"kubernetes.io/projected/f54a589b-695d-45df-b90d-52d4581118d4-kube-api-access-kkkd4\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476210    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-sys\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.476372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.476311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78qh\" (UniqueName: \"kubernetes.io/projected/5ab9927a-f08f-42f7-8668-278ec11cd4d5-kube-api-access-t78qh\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577327 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.577480 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-textfile\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577480 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9927a-f08f-42f7-8668-278ec11cd4d5-metrics-client-ca\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577480 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577480 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f54a589b-695d-45df-b90d-52d4581118d4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-tls\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54a589b-695d-45df-b90d-52d4581118d4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-wtmp\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-root\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkd4\" (UniqueName: \"kubernetes.io/projected/f54a589b-695d-45df-b90d-52d4581118d4-kube-api-access-kkkd4\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-sys\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.577699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78qh\" (UniqueName: \"kubernetes.io/projected/5ab9927a-f08f-42f7-8668-278ec11cd4d5-kube-api-access-t78qh\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-textfile\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577793    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f54a589b-695d-45df-b90d-52d4581118d4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.577831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-root\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.578052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9927a-f08f-42f7-8668-278ec11cd4d5-metrics-client-ca\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.578094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.578147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.578127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-sys\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578524 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.578223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-wtmp\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.578524 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.578379    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54a589b-695d-45df-b90d-52d4581118d4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.578723 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.578695    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.581052 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.581030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.581272 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.581247    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ab9927a-f08f-42f7-8668-278ec11cd4d5-node-exporter-tls\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.581372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.581287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.581573 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.581553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f54a589b-695d-45df-b90d-52d4581118d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.587996 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.587977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkd4\" (UniqueName: \"kubernetes.io/projected/f54a589b-695d-45df-b90d-52d4581118d4-kube-api-access-kkkd4\") pod \"kube-state-metrics-56c5594674-xgn2p\" (UID: \"f54a589b-695d-45df-b90d-52d4581118d4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.588407 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.588388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78qh\" (UniqueName: \"kubernetes.io/projected/5ab9927a-f08f-42f7-8668-278ec11cd4d5-kube-api-access-t78qh\") pod \"node-exporter-7mmpd\" (UID: \"5ab9927a-f08f-42f7-8668-278ec11cd4d5\") " pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:00.687894 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.687808    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"
Jul 23 18:01:00.693916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:00.693889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7mmpd"
Jul 23 18:01:01.586304 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:01.586273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 18:01:01.589207 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:01.589181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecda0a51-585e-43b0-89e0-b5c4e0a52004-metrics-tls\") pod \"dns-default-77vs7\" (UID: \"ecda0a51-585e-43b0-89e0-b5c4e0a52004\") " pod="openshift-dns/dns-default-77vs7"
Jul 23 18:01:01.881690 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:01.881605    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lb24m\""
Jul 23 18:01:01.890048 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:01.890017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77vs7"
Jul 23 18:01:02.025635 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:01:02.025587    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab9927a_f08f_42f7_8668_278ec11cd4d5.slice/crio-fbd44c7dcf92a3746216a43d19edbacbb41b22dbdb885442bfffe112ad978bbd WatchSource:0}: Error finding container fbd44c7dcf92a3746216a43d19edbacbb41b22dbdb885442bfffe112ad978bbd: Status 404 returned error can't find the container with id fbd44c7dcf92a3746216a43d19edbacbb41b22dbdb885442bfffe112ad978bbd
Jul 23 18:01:02.159588 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:02.159555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mmpd" event={"ID":"5ab9927a-f08f-42f7-8668-278ec11cd4d5","Type":"ContainerStarted","Data":"fbd44c7dcf92a3746216a43d19edbacbb41b22dbdb885442bfffe112ad978bbd"}
Jul 23 18:01:02.162148 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:02.162121    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xgn2p"]
Jul 23 18:01:02.165240 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:01:02.165217    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54a589b_695d_45df_b90d_52d4581118d4.slice/crio-8f41a9dfe6b15f1ef579a90250dc47fa16664acfc1ce75916ed189d9b8c46b69 WatchSource:0}: Error finding container 8f41a9dfe6b15f1ef579a90250dc47fa16664acfc1ce75916ed189d9b8c46b69: Status 404 returned error can't find the container with id 8f41a9dfe6b15f1ef579a90250dc47fa16664acfc1ce75916ed189d9b8c46b69
Jul 23 18:01:02.175356 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:02.175332    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-77vs7"]
Jul 23 18:01:02.178324 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:01:02.178303    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecda0a51_585e_43b0_89e0_b5c4e0a52004.slice/crio-60c5109b02ef18bf171a43b0f320e5904088d3f07624290124b549b58ad7d206 WatchSource:0}: Error finding container 60c5109b02ef18bf171a43b0f320e5904088d3f07624290124b549b58ad7d206: Status 404 returned error can't find the container with id 60c5109b02ef18bf171a43b0f320e5904088d3f07624290124b549b58ad7d206
Jul 23 18:01:03.166755 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:03.166716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77vs7" event={"ID":"ecda0a51-585e-43b0-89e0-b5c4e0a52004","Type":"ContainerStarted","Data":"60c5109b02ef18bf171a43b0f320e5904088d3f07624290124b549b58ad7d206"}
Jul 23 18:01:03.169049 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:03.169015    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlnq4" event={"ID":"9bf9f4de-2e4d-4981-a100-9d1c91ba02f5","Type":"ContainerStarted","Data":"994478e4f928135c26f56639000bac8b09b23153f4fe85d6ec5470ff3e9b1c1a"}
Jul 23 18:01:03.170272 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:03.170245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p" event={"ID":"f54a589b-695d-45df-b90d-52d4581118d4","Type":"ContainerStarted","Data":"8f41a9dfe6b15f1ef579a90250dc47fa16664acfc1ce75916ed189d9b8c46b69"}
Jul 23 18:01:03.202136 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:03.202082    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nlnq4" podStartSLOduration=33.807046279 podStartE2EDuration="47.202065168s" totalImagesPullingTime="13.395018889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:00:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:03.202024544 +0000 UTC m=+101.992181659" watchObservedRunningTime="2026-07-23 18:01:03.202065168 +0000 UTC m=+101.992222281"
Jul 23 18:01:04.050815 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.050785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z2zwn"
Jul 23 18:01:04.180346 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.180309    2562 generic.go:356] "Generic (PLEG): container finished" podID="5ab9927a-f08f-42f7-8668-278ec11cd4d5" containerID="4feddb3c52f6d6223d09e67a556f02705ccebf29330bc7a14665eeb009bcf0ea" exitCode=0
Jul 23 18:01:04.180776 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.180402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mmpd" event={"ID":"5ab9927a-f08f-42f7-8668-278ec11cd4d5","Type":"ContainerDied","Data":"4feddb3c52f6d6223d09e67a556f02705ccebf29330bc7a14665eeb009bcf0ea"}
Jul 23 18:01:04.184029 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.183924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p" event={"ID":"f54a589b-695d-45df-b90d-52d4581118d4","Type":"ContainerStarted","Data":"3882f0c0eb3b6f48dce51cb1d1b548e021c33340beee3088791989a518022180"}
Jul 23 18:01:04.184156 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.184063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p" event={"ID":"f54a589b-695d-45df-b90d-52d4581118d4","Type":"ContainerStarted","Data":"257ebaebdd9d85e40533761a94d7c275baf4244aefe6ef2bcb9bfcdefc42c301"}
Jul 23 18:01:04.184156 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.184095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p" event={"ID":"f54a589b-695d-45df-b90d-52d4581118d4","Type":"ContainerStarted","Data":"72cf1b414c34bf79a25b19fcfb65a1c8d60b788e0ad27dfef63aade4380196c9"}
Jul 23 18:01:04.234811 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:04.234757    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-xgn2p" podStartSLOduration=2.9300978090000003 podStartE2EDuration="4.234733897s" totalImagesPullingTime="1.304636088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:04.234476811 +0000 UTC m=+103.024633927" watchObservedRunningTime="2026-07-23 18:01:04.234733897 +0000 UTC m=+103.024891014"
Jul 23 18:01:05.190733 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:05.190686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mmpd" event={"ID":"5ab9927a-f08f-42f7-8668-278ec11cd4d5","Type":"ContainerStarted","Data":"8359e578b5249ee8961af166a1c345960c0686ad594c969303a96cd32a76a115"}
Jul 23 18:01:05.191184 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:05.190753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mmpd" event={"ID":"5ab9927a-f08f-42f7-8668-278ec11cd4d5","Type":"ContainerStarted","Data":"4c73a66b4acb588d4930cf71ecae7ed5b54f39f45f28b8aadbd848afbb9b3351"}
Jul 23 18:01:05.213214 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:05.213163    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7mmpd" podStartSLOduration=3.675334333 podStartE2EDuration="5.213145873s" totalImagesPullingTime="1.445711455s" totalInitContainerRuntime="92.100085ms" isStatefulPod=true podCreationTimestamp="2026-07-23 18:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:05.211572828 +0000 UTC m=+104.001729945" watchObservedRunningTime="2026-07-23 18:01:05.213145873 +0000 UTC m=+104.003302986"
Jul 23 18:01:06.195513 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:06.195466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77vs7" event={"ID":"ecda0a51-585e-43b0-89e0-b5c4e0a52004","Type":"ContainerStarted","Data":"e8c68929eea6ceb79ab2e797fc64642cc568563bb4c5b97d08a410beeb31b30e"}
Jul 23 18:01:06.195905 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:06.195533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77vs7" event={"ID":"ecda0a51-585e-43b0-89e0-b5c4e0a52004","Type":"ContainerStarted","Data":"c923c5f595ee49d88b48392e6722c5b22633d312688ce3c9f9f818e5f6d01efc"}
Jul 23 18:01:06.195905 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:06.195588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-77vs7"
Jul 23 18:01:06.216181 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:06.216127    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-77vs7" podStartSLOduration=65.534153455 podStartE2EDuration="1m9.21611022s" totalImagesPullingTime="3.681956765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:06.215384071 +0000 UTC m=+105.005541188" watchObservedRunningTime="2026-07-23 18:01:06.21611022 +0000 UTC m=+105.006267313"
Jul 23 18:01:16.200619 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:16.200589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-77vs7"
Jul 23 18:01:21.249523 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:21.249484    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1abe68d-f131-43c3-86b5-224632d4bd30" containerID="6afd93a59a0c1ac1933466cf427c7b44a49f726862e368c9ec7c2a30058e06b1" exitCode=0
Jul 23 18:01:21.249884 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:21.249530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-f72dk" event={"ID":"d1abe68d-f131-43c3-86b5-224632d4bd30","Type":"ContainerDied","Data":"6afd93a59a0c1ac1933466cf427c7b44a49f726862e368c9ec7c2a30058e06b1"}
Jul 23 18:01:21.249884 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:21.249877    2562 scope.go:117] "RemoveContainer" containerID="6afd93a59a0c1ac1933466cf427c7b44a49f726862e368c9ec7c2a30058e06b1"
Jul 23 18:01:22.176428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:22.176398    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77vs7_ecda0a51-585e-43b0-89e0-b5c4e0a52004/dns/0.log"
Jul 23 18:01:22.254919 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:22.254886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-f72dk" event={"ID":"d1abe68d-f131-43c3-86b5-224632d4bd30","Type":"ContainerStarted","Data":"f900ff6def3747857e18d87fb3413211e40dbebf230d8f3aa99143add6427de4"}
Jul 23 18:01:22.376180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:22.376153    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77vs7_ecda0a51-585e-43b0-89e0-b5c4e0a52004/kube-rbac-proxy/0.log"
Jul 23 18:01:23.376088 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:23.376061    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9xbxt_94942e6d-c9a9-4fb5-bac0-4aa81845e979/dns-node-resolver/0.log"
Jul 23 18:01:23.977111 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:23.977086    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7cc9cdfd-b5f4s_aab248f1-1f0e-4550-abfb-ccf8deff11e6/router/0.log"
Jul 23 18:01:24.576297 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:24.576273    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhlzz_cd24080b-11ac-4d1a-85c5-f3928b81d06e/serve-healthcheck-canary/0.log"
Jul 23 18:01:31.415849 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:31.415798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 18:01:31.418314 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:31.418288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197e0944-75f2-4c47-8ed0-53194766062d-metrics-certs\") pod \"network-metrics-daemon-vd8gk\" (UID: \"197e0944-75f2-4c47-8ed0-53194766062d\") " pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 18:01:31.716398 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:31.716317    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hv4kc\""
Jul 23 18:01:31.724168 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:31.724147    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vd8gk"
Jul 23 18:01:31.892910 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:31.892859    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vd8gk"]
Jul 23 18:01:31.897603 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:01:31.897569    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197e0944_75f2_4c47_8ed0_53194766062d.slice/crio-82d1a7f708935d8d4fa099ecc0845168d402c93b42f469e58927e40c84c8b793 WatchSource:0}: Error finding container 82d1a7f708935d8d4fa099ecc0845168d402c93b42f469e58927e40c84c8b793: Status 404 returned error can't find the container with id 82d1a7f708935d8d4fa099ecc0845168d402c93b42f469e58927e40c84c8b793
Jul 23 18:01:32.290197 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:32.290160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vd8gk" event={"ID":"197e0944-75f2-4c47-8ed0-53194766062d","Type":"ContainerStarted","Data":"82d1a7f708935d8d4fa099ecc0845168d402c93b42f469e58927e40c84c8b793"}
Jul 23 18:01:33.298557 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:33.298436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vd8gk" event={"ID":"197e0944-75f2-4c47-8ed0-53194766062d","Type":"ContainerStarted","Data":"de3b8c28d4f08099e79c2ace041df8f81ce397eca569446c752a1bf3c15aee99"}
Jul 23 18:01:33.298557 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:33.298496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vd8gk" event={"ID":"197e0944-75f2-4c47-8ed0-53194766062d","Type":"ContainerStarted","Data":"3513a7d821d3c36ee855ef22f455cc35d9994fbbab4f95eb13e66f3500a1240e"}
Jul 23 18:01:33.318394 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:01:33.318340    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vd8gk" podStartSLOduration=130.146134899 podStartE2EDuration="2m11.318322771s" totalImagesPullingTime="1.172187872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:33.31729079 +0000 UTC m=+132.107447925" watchObservedRunningTime="2026-07-23 18:01:33.318322771 +0000 UTC m=+132.108479886"
Jul 23 18:04:21.514193 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:04:21.514166    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:04:21.516954 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:04:21.516919    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:04:21.519604 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:04:21.519584    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 18:05:41.946379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:41.946299    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"]
Jul 23 18:05:41.949519 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:41.949504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:41.954878 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:41.954856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 23 18:05:41.955020 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:41.954857    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 23 18:05:41.955682 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:41.955584    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2fhzr\""
Jul 23 18:05:41.986202 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:41.986166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"]
Jul 23 18:05:42.038334 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.038296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.038334 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.038334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bpd\" (UniqueName: \"kubernetes.io/projected/c40b3b6c-45e4-4724-a2e4-57771384df41-kube-api-access-z6bpd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.038570 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.038375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.138733 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.138709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.138733 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.138739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bpd\" (UniqueName: \"kubernetes.io/projected/c40b3b6c-45e4-4724-a2e4-57771384df41-kube-api-access-z6bpd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.138910 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.138763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.139124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.139108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.139187 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.139170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.149934 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.149909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bpd\" (UniqueName: \"kubernetes.io/projected/c40b3b6c-45e4-4724-a2e4-57771384df41-kube-api-access-z6bpd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.258095 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.258068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:42.399393 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.399360    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"]
Jul 23 18:05:42.402346 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:05:42.402315    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40b3b6c_45e4_4724_a2e4_57771384df41.slice/crio-0f3e4ec2785d664da5a2dfd71aa020e110f239027a83649dd6164b80a3d688e9 WatchSource:0}: Error finding container 0f3e4ec2785d664da5a2dfd71aa020e110f239027a83649dd6164b80a3d688e9: Status 404 returned error can't find the container with id 0f3e4ec2785d664da5a2dfd71aa020e110f239027a83649dd6164b80a3d688e9
Jul 23 18:05:42.404213 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:42.404195    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:05:43.137211 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:43.137177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4" event={"ID":"c40b3b6c-45e4-4724-a2e4-57771384df41","Type":"ContainerStarted","Data":"0f3e4ec2785d664da5a2dfd71aa020e110f239027a83649dd6164b80a3d688e9"}
Jul 23 18:05:48.157828 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:48.157787    2562 generic.go:356] "Generic (PLEG): container finished" podID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerID="7ce023652208b29ad6481d862369cd7daf65055c406525d92345de6d60fe2312" exitCode=0
Jul 23 18:05:48.158250 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:48.157842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4" event={"ID":"c40b3b6c-45e4-4724-a2e4-57771384df41","Type":"ContainerDied","Data":"7ce023652208b29ad6481d862369cd7daf65055c406525d92345de6d60fe2312"}
Jul 23 18:05:51.171685 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:51.171651    2562 generic.go:356] "Generic (PLEG): container finished" podID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerID="02ec47141ea6b377cce3600fd62488ba3c6f01d8d3901f69d251ddcd2b32100e" exitCode=0
Jul 23 18:05:51.172128 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:51.171731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4" event={"ID":"c40b3b6c-45e4-4724-a2e4-57771384df41","Type":"ContainerDied","Data":"02ec47141ea6b377cce3600fd62488ba3c6f01d8d3901f69d251ddcd2b32100e"}
Jul 23 18:05:58.199608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:58.199569    2562 generic.go:356] "Generic (PLEG): container finished" podID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerID="83f593ccced8412e110d6ef6083374a87e42c05930dafb872fbaef5c122d6a4c" exitCode=0
Jul 23 18:05:58.200006 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:58.199647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4" event={"ID":"c40b3b6c-45e4-4724-a2e4-57771384df41","Type":"ContainerDied","Data":"83f593ccced8412e110d6ef6083374a87e42c05930dafb872fbaef5c122d6a4c"}
Jul 23 18:05:59.322694 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.322672    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:05:59.476638 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.476552    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bpd\" (UniqueName: \"kubernetes.io/projected/c40b3b6c-45e4-4724-a2e4-57771384df41-kube-api-access-z6bpd\") pod \"c40b3b6c-45e4-4724-a2e4-57771384df41\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") "
Jul 23 18:05:59.476638 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.476635    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-bundle\") pod \"c40b3b6c-45e4-4724-a2e4-57771384df41\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") "
Jul 23 18:05:59.476866 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.476669    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-util\") pod \"c40b3b6c-45e4-4724-a2e4-57771384df41\" (UID: \"c40b3b6c-45e4-4724-a2e4-57771384df41\") "
Jul 23 18:05:59.477314 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.477288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-bundle" (OuterVolumeSpecName: "bundle") pod "c40b3b6c-45e4-4724-a2e4-57771384df41" (UID: "c40b3b6c-45e4-4724-a2e4-57771384df41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:05:59.478937 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.478908    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40b3b6c-45e4-4724-a2e4-57771384df41-kube-api-access-z6bpd" (OuterVolumeSpecName: "kube-api-access-z6bpd") pod "c40b3b6c-45e4-4724-a2e4-57771384df41" (UID: "c40b3b6c-45e4-4724-a2e4-57771384df41"). InnerVolumeSpecName "kube-api-access-z6bpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:05:59.481836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.481812    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-util" (OuterVolumeSpecName: "util") pod "c40b3b6c-45e4-4724-a2e4-57771384df41" (UID: "c40b3b6c-45e4-4724-a2e4-57771384df41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:05:59.577631 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.577579    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:05:59.577631 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.577624    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b3b6c-45e4-4724-a2e4-57771384df41-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:05:59.577631 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:05:59.577633    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bpd\" (UniqueName: \"kubernetes.io/projected/c40b3b6c-45e4-4724-a2e4-57771384df41-kube-api-access-z6bpd\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:00.212043 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:00.212016    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4"
Jul 23 18:06:00.212043 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:00.212024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkxz4" event={"ID":"c40b3b6c-45e4-4724-a2e4-57771384df41","Type":"ContainerDied","Data":"0f3e4ec2785d664da5a2dfd71aa020e110f239027a83649dd6164b80a3d688e9"}
Jul 23 18:06:00.212247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:00.212087    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f3e4ec2785d664da5a2dfd71aa020e110f239027a83649dd6164b80a3d688e9"
Jul 23 18:06:10.716233 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716195    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"]
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716538    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="util"
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716555    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="util"
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716579    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="pull"
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716586    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="pull"
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716595    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="extract"
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716602    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="extract"
Jul 23 18:06:10.716706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.716672    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40b3b6c-45e4-4724-a2e4-57771384df41" containerName="extract"
Jul 23 18:06:10.719972 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.719933    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.726188 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.726168    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 23 18:06:10.727662 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.727640    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2fhzr\""
Jul 23 18:06:10.728043 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.728030    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 23 18:06:10.742608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.742586    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"]
Jul 23 18:06:10.856760 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.856725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.856967 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.856777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljcj\" (UniqueName: \"kubernetes.io/projected/4bef090b-d2e0-4b20-97ea-f5a6067958c7-kube-api-access-qljcj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.856967 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.856846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.957744 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.957690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.957973 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.957766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljcj\" (UniqueName: \"kubernetes.io/projected/4bef090b-d2e0-4b20-97ea-f5a6067958c7-kube-api-access-qljcj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.957973 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.957804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.958120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.958078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.958172 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.958154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:10.975513 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:10.975444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljcj\" (UniqueName: \"kubernetes.io/projected/4bef090b-d2e0-4b20-97ea-f5a6067958c7-kube-api-access-qljcj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:11.028317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:11.028279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:11.166036 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:11.165869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"]
Jul 23 18:06:11.171183 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:06:11.171155    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bef090b_d2e0_4b20_97ea_f5a6067958c7.slice/crio-a3667a08b3b8d2c58c8960b9e3dcef611a9d28fbeb509b86f94be1cf367db1ce WatchSource:0}: Error finding container a3667a08b3b8d2c58c8960b9e3dcef611a9d28fbeb509b86f94be1cf367db1ce: Status 404 returned error can't find the container with id a3667a08b3b8d2c58c8960b9e3dcef611a9d28fbeb509b86f94be1cf367db1ce
Jul 23 18:06:11.254363 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:11.254328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b" event={"ID":"4bef090b-d2e0-4b20-97ea-f5a6067958c7","Type":"ContainerStarted","Data":"4cf50d819f656035057e1b2cb1b5d171d06a102cffa68535d5de7d82dcc76794"}
Jul 23 18:06:11.254521 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:11.254384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b" event={"ID":"4bef090b-d2e0-4b20-97ea-f5a6067958c7","Type":"ContainerStarted","Data":"a3667a08b3b8d2c58c8960b9e3dcef611a9d28fbeb509b86f94be1cf367db1ce"}
Jul 23 18:06:12.259293 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:12.259261    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerID="4cf50d819f656035057e1b2cb1b5d171d06a102cffa68535d5de7d82dcc76794" exitCode=0
Jul 23 18:06:12.259657 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:12.259337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b" event={"ID":"4bef090b-d2e0-4b20-97ea-f5a6067958c7","Type":"ContainerDied","Data":"4cf50d819f656035057e1b2cb1b5d171d06a102cffa68535d5de7d82dcc76794"}
Jul 23 18:06:14.268197 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:14.268163    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerID="85625f9c8568e0913a89880cae2ab3f97c217d9e62a37ef0c51c96a4dc329519" exitCode=0
Jul 23 18:06:14.268664 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:14.268216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b" event={"ID":"4bef090b-d2e0-4b20-97ea-f5a6067958c7","Type":"ContainerDied","Data":"85625f9c8568e0913a89880cae2ab3f97c217d9e62a37ef0c51c96a4dc329519"}
Jul 23 18:06:15.274228 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:15.274188    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerID="75bed6bf7aa95b0a133d16143aaeb0eab60ca5a20a0f498f86a59e2c56a28c3c" exitCode=0
Jul 23 18:06:15.274637 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:15.274278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b" event={"ID":"4bef090b-d2e0-4b20-97ea-f5a6067958c7","Type":"ContainerDied","Data":"75bed6bf7aa95b0a133d16143aaeb0eab60ca5a20a0f498f86a59e2c56a28c3c"}
Jul 23 18:06:16.395968 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.395929    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:16.496851 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.496813    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qljcj\" (UniqueName: \"kubernetes.io/projected/4bef090b-d2e0-4b20-97ea-f5a6067958c7-kube-api-access-qljcj\") pod \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") "
Jul 23 18:06:16.497057 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.496887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-util\") pod \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") "
Jul 23 18:06:16.497057 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.496909    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-bundle\") pod \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\" (UID: \"4bef090b-d2e0-4b20-97ea-f5a6067958c7\") "
Jul 23 18:06:16.497345 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.497317    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-bundle" (OuterVolumeSpecName: "bundle") pod "4bef090b-d2e0-4b20-97ea-f5a6067958c7" (UID: "4bef090b-d2e0-4b20-97ea-f5a6067958c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:06:16.499146 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.499122    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bef090b-d2e0-4b20-97ea-f5a6067958c7-kube-api-access-qljcj" (OuterVolumeSpecName: "kube-api-access-qljcj") pod "4bef090b-d2e0-4b20-97ea-f5a6067958c7" (UID: "4bef090b-d2e0-4b20-97ea-f5a6067958c7"). InnerVolumeSpecName "kube-api-access-qljcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:06:16.502185 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.502159    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-util" (OuterVolumeSpecName: "util") pod "4bef090b-d2e0-4b20-97ea-f5a6067958c7" (UID: "4bef090b-d2e0-4b20-97ea-f5a6067958c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:06:16.597486 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.597393    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:16.597486 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.597423    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bef090b-d2e0-4b20-97ea-f5a6067958c7-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:16.597486 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:16.597431    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljcj\" (UniqueName: \"kubernetes.io/projected/4bef090b-d2e0-4b20-97ea-f5a6067958c7-kube-api-access-qljcj\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:17.283700 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:17.283658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b" event={"ID":"4bef090b-d2e0-4b20-97ea-f5a6067958c7","Type":"ContainerDied","Data":"a3667a08b3b8d2c58c8960b9e3dcef611a9d28fbeb509b86f94be1cf367db1ce"}
Jul 23 18:06:17.283700 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:17.283706    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzc74b"
Jul 23 18:06:17.283913 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:17.283720    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3667a08b3b8d2c58c8960b9e3dcef611a9d28fbeb509b86f94be1cf367db1ce"
Jul 23 18:06:28.397902 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.397870    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"]
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398159    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="util"
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398170    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="util"
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398180    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="pull"
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398185    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="pull"
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398192    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="extract"
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398198    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="extract"
Jul 23 18:06:28.398262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.398241    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bef090b-d2e0-4b20-97ea-f5a6067958c7" containerName="extract"
Jul 23 18:06:28.455132 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.455101    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"]
Jul 23 18:06:28.455299 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.455213    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.458048 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.458027    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 23 18:06:28.459769 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.459752    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 23 18:06:28.459880 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.459757    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2fhzr\""
Jul 23 18:06:28.489427 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.489398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.489560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.489464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.489560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.489491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7c8\" (UniqueName: \"kubernetes.io/projected/215da959-c148-4a68-8cf9-9a798dd6ce89-kube-api-access-4w7c8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.590765 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.590731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.590904 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.590769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7c8\" (UniqueName: \"kubernetes.io/projected/215da959-c148-4a68-8cf9-9a798dd6ce89-kube-api-access-4w7c8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.590904 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.590814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.591227 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.591204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.591266 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.591218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.600936 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.600911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7c8\" (UniqueName: \"kubernetes.io/projected/215da959-c148-4a68-8cf9-9a798dd6ce89-kube-api-access-4w7c8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.764047 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.764013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:28.913395 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:28.913371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"]
Jul 23 18:06:28.916147 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:06:28.916117    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215da959_c148_4a68_8cf9_9a798dd6ce89.slice/crio-cf9704fb5bb3cacd3135d06a7233210b70d2d757fb3a3a2ec63101232d4c2b8b WatchSource:0}: Error finding container cf9704fb5bb3cacd3135d06a7233210b70d2d757fb3a3a2ec63101232d4c2b8b: Status 404 returned error can't find the container with id cf9704fb5bb3cacd3135d06a7233210b70d2d757fb3a3a2ec63101232d4c2b8b
Jul 23 18:06:29.329418 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:29.329322    2562 generic.go:356] "Generic (PLEG): container finished" podID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerID="2ecd829f2ae3888ee324bbaa93b7c94d7c6f9892e7f74708624f3f6267afc49c" exitCode=0
Jul 23 18:06:29.329577 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:29.329410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5" event={"ID":"215da959-c148-4a68-8cf9-9a798dd6ce89","Type":"ContainerDied","Data":"2ecd829f2ae3888ee324bbaa93b7c94d7c6f9892e7f74708624f3f6267afc49c"}
Jul 23 18:06:29.329577 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:29.329469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5" event={"ID":"215da959-c148-4a68-8cf9-9a798dd6ce89","Type":"ContainerStarted","Data":"cf9704fb5bb3cacd3135d06a7233210b70d2d757fb3a3a2ec63101232d4c2b8b"}
Jul 23 18:06:32.341584 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:32.341550    2562 generic.go:356] "Generic (PLEG): container finished" podID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerID="41733576482a6c972864339d1d5c4811029b6b9b0772a34265615cd7c105c594" exitCode=0
Jul 23 18:06:32.342013 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:32.341623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5" event={"ID":"215da959-c148-4a68-8cf9-9a798dd6ce89","Type":"ContainerDied","Data":"41733576482a6c972864339d1d5c4811029b6b9b0772a34265615cd7c105c594"}
Jul 23 18:06:33.347581 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:33.347545    2562 generic.go:356] "Generic (PLEG): container finished" podID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerID="e85b156313fb4f7f9a1edd39766744a2009fd7000680a13a2577ed510d162866" exitCode=0
Jul 23 18:06:33.347973 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:33.347616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5" event={"ID":"215da959-c148-4a68-8cf9-9a798dd6ce89","Type":"ContainerDied","Data":"e85b156313fb4f7f9a1edd39766744a2009fd7000680a13a2577ed510d162866"}
Jul 23 18:06:34.467416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.467394    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:34.535036 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.534999    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-util\") pod \"215da959-c148-4a68-8cf9-9a798dd6ce89\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") "
Jul 23 18:06:34.535036 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.535039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-bundle\") pod \"215da959-c148-4a68-8cf9-9a798dd6ce89\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") "
Jul 23 18:06:34.535280 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.535066    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w7c8\" (UniqueName: \"kubernetes.io/projected/215da959-c148-4a68-8cf9-9a798dd6ce89-kube-api-access-4w7c8\") pod \"215da959-c148-4a68-8cf9-9a798dd6ce89\" (UID: \"215da959-c148-4a68-8cf9-9a798dd6ce89\") "
Jul 23 18:06:34.535747 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.535722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-bundle" (OuterVolumeSpecName: "bundle") pod "215da959-c148-4a68-8cf9-9a798dd6ce89" (UID: "215da959-c148-4a68-8cf9-9a798dd6ce89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:06:34.537298 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.537271    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215da959-c148-4a68-8cf9-9a798dd6ce89-kube-api-access-4w7c8" (OuterVolumeSpecName: "kube-api-access-4w7c8") pod "215da959-c148-4a68-8cf9-9a798dd6ce89" (UID: "215da959-c148-4a68-8cf9-9a798dd6ce89"). InnerVolumeSpecName "kube-api-access-4w7c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:06:34.600686 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.600620    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-util" (OuterVolumeSpecName: "util") pod "215da959-c148-4a68-8cf9-9a798dd6ce89" (UID: "215da959-c148-4a68-8cf9-9a798dd6ce89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:06:34.636279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.636207    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:34.636279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.636238    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/215da959-c148-4a68-8cf9-9a798dd6ce89-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:34.636279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:34.636251    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w7c8\" (UniqueName: \"kubernetes.io/projected/215da959-c148-4a68-8cf9-9a798dd6ce89-kube-api-access-4w7c8\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:06:35.357181 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:35.357149    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5"
Jul 23 18:06:35.357359 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:35.357149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftppf5" event={"ID":"215da959-c148-4a68-8cf9-9a798dd6ce89","Type":"ContainerDied","Data":"cf9704fb5bb3cacd3135d06a7233210b70d2d757fb3a3a2ec63101232d4c2b8b"}
Jul 23 18:06:35.357359 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:35.357261    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf9704fb5bb3cacd3135d06a7233210b70d2d757fb3a3a2ec63101232d4c2b8b"
Jul 23 18:06:46.203995 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.203909    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"]
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204158    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="pull"
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204169    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="pull"
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204180    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="util"
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204186    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="util"
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204196    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="extract"
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204202    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="extract"
Jul 23 18:06:46.204435 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.204244    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="215da959-c148-4a68-8cf9-9a798dd6ce89" containerName="extract"
Jul 23 18:06:46.206921 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.206904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.209822 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.209798    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 18:06:46.210088 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.210070    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g6dq5\""
Jul 23 18:06:46.210314 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.210299    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 18:06:46.210370 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.210324    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 18:06:46.210602 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.210587    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 18:06:46.229875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.229844    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"]
Jul 23 18:06:46.327250 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.327213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6619904-ed33-46f9-8c3c-95429e190ea9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.327250 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.327247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6619904-ed33-46f9-8c3c-95429e190ea9-tmp\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.327449 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.327274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6619904-ed33-46f9-8c3c-95429e190ea9-manifests\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.327449 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.327347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsfb2\" (UniqueName: \"kubernetes.io/projected/f6619904-ed33-46f9-8c3c-95429e190ea9-kube-api-access-bsfb2\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.327449 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.327425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6619904-ed33-46f9-8c3c-95429e190ea9-webhook-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.428338 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.428293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6619904-ed33-46f9-8c3c-95429e190ea9-webhook-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.428338 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.428347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6619904-ed33-46f9-8c3c-95429e190ea9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.428575 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.428369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6619904-ed33-46f9-8c3c-95429e190ea9-tmp\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.428575 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.428408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6619904-ed33-46f9-8c3c-95429e190ea9-manifests\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.428575 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.428435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsfb2\" (UniqueName: \"kubernetes.io/projected/f6619904-ed33-46f9-8c3c-95429e190ea9-kube-api-access-bsfb2\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.428855 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.428830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6619904-ed33-46f9-8c3c-95429e190ea9-manifests\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.430822 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.430798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6619904-ed33-46f9-8c3c-95429e190ea9-tmp\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.430916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.430840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6619904-ed33-46f9-8c3c-95429e190ea9-webhook-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.430916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.430903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6619904-ed33-46f9-8c3c-95429e190ea9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.468216 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.468149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsfb2\" (UniqueName: \"kubernetes.io/projected/f6619904-ed33-46f9-8c3c-95429e190ea9-kube-api-access-bsfb2\") pod \"opendatahub-operator-controller-manager-657dbb4b4c-wqmd5\" (UID: \"f6619904-ed33-46f9-8c3c-95429e190ea9\") " pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.516343 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.516309    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:46.656439 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:46.656401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"]
Jul 23 18:06:46.658635 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:06:46.658611    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6619904_ed33_46f9_8c3c_95429e190ea9.slice/crio-08907524f13f6086481166756bf43dc0f8ae20d796373c10858a4e9b707783eb WatchSource:0}: Error finding container 08907524f13f6086481166756bf43dc0f8ae20d796373c10858a4e9b707783eb: Status 404 returned error can't find the container with id 08907524f13f6086481166756bf43dc0f8ae20d796373c10858a4e9b707783eb
Jul 23 18:06:47.404170 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:47.404137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5" event={"ID":"f6619904-ed33-46f9-8c3c-95429e190ea9","Type":"ContainerStarted","Data":"08907524f13f6086481166756bf43dc0f8ae20d796373c10858a4e9b707783eb"}
Jul 23 18:06:50.417570 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:50.417531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5" event={"ID":"f6619904-ed33-46f9-8c3c-95429e190ea9","Type":"ContainerStarted","Data":"648135d510c911d9cddeb2ae2018027e45e3b174d6c5a6dd978f7ba8f410e538"}
Jul 23 18:06:51.422678 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:51.422642    2562 generic.go:356] "Generic (PLEG): container finished" podID="f6619904-ed33-46f9-8c3c-95429e190ea9" containerID="648135d510c911d9cddeb2ae2018027e45e3b174d6c5a6dd978f7ba8f410e538" exitCode=0
Jul 23 18:06:51.423075 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:51.422717    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5" event={"ID":"f6619904-ed33-46f9-8c3c-95429e190ea9","Type":"ContainerDied","Data":"648135d510c911d9cddeb2ae2018027e45e3b174d6c5a6dd978f7ba8f410e538"}
Jul 23 18:06:52.428584 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:52.428545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5" event={"ID":"f6619904-ed33-46f9-8c3c-95429e190ea9","Type":"ContainerStarted","Data":"8c2f5b0dec4b8174e1e44fdd9925eb158c1c877a71f06fdbfe1b77bf25dc0adb"}
Jul 23 18:06:52.428962 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:52.428694    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:06:52.456164 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:06:52.456106    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5" podStartSLOduration=1.954522329 podStartE2EDuration="6.456087505s" totalImagesPullingTime="3.422350272s" totalInitContainerRuntime="1.079214904s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:06:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:52.454549123 +0000 UTC m=+451.244706251" watchObservedRunningTime="2026-07-23 18:06:52.456087505 +0000 UTC m=+451.246244635"
Jul 23 18:07:03.435025 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:03.434988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-657dbb4b4c-wqmd5"
Jul 23 18:07:06.864985 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.864937    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"]
Jul 23 18:07:06.872981 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.872955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:06.877260 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.877240    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 18:07:06.877549 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.877528    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 18:07:06.900922 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.900889    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"]
Jul 23 18:07:06.995041 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.995012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd75b15-abb2-404b-9978-7e1252260b6e-tls-certs\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:06.995041 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.995045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5nt\" (UniqueName: \"kubernetes.io/projected/ffd75b15-abb2-404b-9978-7e1252260b6e-kube-api-access-gl5nt\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:06.995249 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:06.995085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd75b15-abb2-404b-9978-7e1252260b6e-tmp\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.027588 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.027561    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jg4bl"]
Jul 23 18:07:07.030827 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.030811    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.034168 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.034151    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qfjr9\""
Jul 23 18:07:07.034221 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.034152    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 23 18:07:07.074120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.074090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jg4bl"]
Jul 23 18:07:07.095748 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.095715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd75b15-abb2-404b-9978-7e1252260b6e-tls-certs\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.095748 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.095747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5nt\" (UniqueName: \"kubernetes.io/projected/ffd75b15-abb2-404b-9978-7e1252260b6e-kube-api-access-gl5nt\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.095995 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.095790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd75b15-abb2-404b-9978-7e1252260b6e-tmp\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.095995 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.095840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqsm\" (UniqueName: \"kubernetes.io/projected/837026c8-64fa-4afa-b45c-3bbb0d092487-kube-api-access-9fqsm\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.095995 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.095867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/837026c8-64fa-4afa-b45c-3bbb0d092487-cert\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.098247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.098218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd75b15-abb2-404b-9978-7e1252260b6e-tmp\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.098428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.098412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd75b15-abb2-404b-9978-7e1252260b6e-tls-certs\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.117193 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.117134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5nt\" (UniqueName: \"kubernetes.io/projected/ffd75b15-abb2-404b-9978-7e1252260b6e-kube-api-access-gl5nt\") pod \"kube-auth-proxy-77d9cbf8b4-wmqzj\" (UID: \"ffd75b15-abb2-404b-9978-7e1252260b6e\") " pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.183806 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.183773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"
Jul 23 18:07:07.196894 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.196869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqsm\" (UniqueName: \"kubernetes.io/projected/837026c8-64fa-4afa-b45c-3bbb0d092487-kube-api-access-9fqsm\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.197027 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.196910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/837026c8-64fa-4afa-b45c-3bbb0d092487-cert\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.197099 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:07:07.197078    2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 23 18:07:07.197185 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:07:07.197154    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837026c8-64fa-4afa-b45c-3bbb0d092487-cert podName:837026c8-64fa-4afa-b45c-3bbb0d092487 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:07.697132331 +0000 UTC m=+466.487289442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/837026c8-64fa-4afa-b45c-3bbb0d092487-cert") pod "odh-model-controller-7998c67dc4-jg4bl" (UID: "837026c8-64fa-4afa-b45c-3bbb0d092487") : secret "odh-model-controller-webhook-cert" not found
Jul 23 18:07:07.216785 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.216749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqsm\" (UniqueName: \"kubernetes.io/projected/837026c8-64fa-4afa-b45c-3bbb0d092487-kube-api-access-9fqsm\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.379243 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.379219    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj"]
Jul 23 18:07:07.379852 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:07:07.379834    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd75b15_abb2_404b_9978_7e1252260b6e.slice/crio-fa362477e6ffb932b51b6cb9f3459f44696610c3e10586e5e5faa7187456f1da WatchSource:0}: Error finding container fa362477e6ffb932b51b6cb9f3459f44696610c3e10586e5e5faa7187456f1da: Status 404 returned error can't find the container with id fa362477e6ffb932b51b6cb9f3459f44696610c3e10586e5e5faa7187456f1da
Jul 23 18:07:07.486347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.486311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj" event={"ID":"ffd75b15-abb2-404b-9978-7e1252260b6e","Type":"ContainerStarted","Data":"fa362477e6ffb932b51b6cb9f3459f44696610c3e10586e5e5faa7187456f1da"}
Jul 23 18:07:07.701351 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.701258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/837026c8-64fa-4afa-b45c-3bbb0d092487-cert\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.703702 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.703678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/837026c8-64fa-4afa-b45c-3bbb0d092487-cert\") pod \"odh-model-controller-7998c67dc4-jg4bl\" (UID: \"837026c8-64fa-4afa-b45c-3bbb0d092487\") " pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:07.940523 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:07.940489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:08.077850 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.077820    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jg4bl"]
Jul 23 18:07:08.081268 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:07:08.081237    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837026c8_64fa_4afa_b45c_3bbb0d092487.slice/crio-93e0a86bebd04062b3bda3868054873c83d94d9d5d6941819f26458734b2a806 WatchSource:0}: Error finding container 93e0a86bebd04062b3bda3868054873c83d94d9d5d6941819f26458734b2a806: Status 404 returned error can't find the container with id 93e0a86bebd04062b3bda3868054873c83d94d9d5d6941819f26458734b2a806
Jul 23 18:07:08.491890 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.491857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl" event={"ID":"837026c8-64fa-4afa-b45c-3bbb0d092487","Type":"ContainerStarted","Data":"93e0a86bebd04062b3bda3868054873c83d94d9d5d6941819f26458734b2a806"}
Jul 23 18:07:08.590924 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.590886    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gngw7"]
Jul 23 18:07:08.600389 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.600351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.603687 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.603668    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 23 18:07:08.620330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.620301    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gngw7"]
Jul 23 18:07:08.711317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.711282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9746c\" (UniqueName: \"kubernetes.io/projected/a79a2916-00f0-47b2-b9a2-60bd57c5b596-kube-api-access-9746c\") pod \"model-serving-api-54d89bd5fc-gngw7\" (UID: \"a79a2916-00f0-47b2-b9a2-60bd57c5b596\") " pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.711317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.711321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79a2916-00f0-47b2-b9a2-60bd57c5b596-tls-certs\") pod \"model-serving-api-54d89bd5fc-gngw7\" (UID: \"a79a2916-00f0-47b2-b9a2-60bd57c5b596\") " pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.812582 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.812541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9746c\" (UniqueName: \"kubernetes.io/projected/a79a2916-00f0-47b2-b9a2-60bd57c5b596-kube-api-access-9746c\") pod \"model-serving-api-54d89bd5fc-gngw7\" (UID: \"a79a2916-00f0-47b2-b9a2-60bd57c5b596\") " pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.812788 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.812589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79a2916-00f0-47b2-b9a2-60bd57c5b596-tls-certs\") pod \"model-serving-api-54d89bd5fc-gngw7\" (UID: \"a79a2916-00f0-47b2-b9a2-60bd57c5b596\") " pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.815264 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.815236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79a2916-00f0-47b2-b9a2-60bd57c5b596-tls-certs\") pod \"model-serving-api-54d89bd5fc-gngw7\" (UID: \"a79a2916-00f0-47b2-b9a2-60bd57c5b596\") " pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.831019 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.830990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9746c\" (UniqueName: \"kubernetes.io/projected/a79a2916-00f0-47b2-b9a2-60bd57c5b596-kube-api-access-9746c\") pod \"model-serving-api-54d89bd5fc-gngw7\" (UID: \"a79a2916-00f0-47b2-b9a2-60bd57c5b596\") " pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:08.910221 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:08.910181    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:09.088861 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.088831    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gngw7"]
Jul 23 18:07:09.091606 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:07:09.091567    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79a2916_00f0_47b2_b9a2_60bd57c5b596.slice/crio-ab850bb504693c19b7dd6c67ffcc4b956defc77021f0d770d45448c6bdc02eea WatchSource:0}: Error finding container ab850bb504693c19b7dd6c67ffcc4b956defc77021f0d770d45448c6bdc02eea: Status 404 returned error can't find the container with id ab850bb504693c19b7dd6c67ffcc4b956defc77021f0d770d45448c6bdc02eea
Jul 23 18:07:09.199730 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.199693    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"]
Jul 23 18:07:09.204716 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.204689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.207545 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.207519    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 18:07:09.207767 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.207732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 18:07:09.208050 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.208034    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 18:07:09.208245 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.208226    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sxwzx\""
Jul 23 18:07:09.208491 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.208472    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 18:07:09.227248 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.227202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"]
Jul 23 18:07:09.317490 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.317490 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-cacerts\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.317718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.317718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb4112-e048-45e4-b25f-369eae3d9c35-local-certs\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.317718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.317718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zzw\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-kube-api-access-r6zzw\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.317718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.317618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-token\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.418968 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.418866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.418968 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.418902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb4112-e048-45e4-b25f-369eae3d9c35-local-certs\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.418968 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.418921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.418968 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.418965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zzw\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-kube-api-access-r6zzw\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.419337 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.419019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-token\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.419337 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.419071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.419337 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.419204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-cacerts\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.419747 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.419668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.421458 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.421433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb4112-e048-45e4-b25f-369eae3d9c35-local-certs\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.421659 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.421628    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.422084 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.422061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-cacerts\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.422164 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.422136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.433339 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.433315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zzw\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-kube-api-access-r6zzw\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.433450 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.433371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-token\") pod \"istiod-openshift-gateway-7574459656-5ch84\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:09.499653 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.499612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7" event={"ID":"a79a2916-00f0-47b2-b9a2-60bd57c5b596","Type":"ContainerStarted","Data":"ab850bb504693c19b7dd6c67ffcc4b956defc77021f0d770d45448c6bdc02eea"}
Jul 23 18:07:09.517188 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:09.517127    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:10.470754 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:10.470618    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"]
Jul 23 18:07:10.473839 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:07:10.473803    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafeb4112_e048_45e4_b25f_369eae3d9c35.slice/crio-f3f1073efe81d1691c6b85d8eec60300926e656b19e3f1bfd685e933f871fcd4 WatchSource:0}: Error finding container f3f1073efe81d1691c6b85d8eec60300926e656b19e3f1bfd685e933f871fcd4: Status 404 returned error can't find the container with id f3f1073efe81d1691c6b85d8eec60300926e656b19e3f1bfd685e933f871fcd4
Jul 23 18:07:10.506628 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:10.506591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj" event={"ID":"ffd75b15-abb2-404b-9978-7e1252260b6e","Type":"ContainerStarted","Data":"8361f68a823d837ec1b9839727b0f189029642a1e522ff957806fc10cc26779e"}
Jul 23 18:07:10.507913 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:10.507886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" event={"ID":"afeb4112-e048-45e4-b25f-369eae3d9c35","Type":"ContainerStarted","Data":"f3f1073efe81d1691c6b85d8eec60300926e656b19e3f1bfd685e933f871fcd4"}
Jul 23 18:07:10.530898 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:10.530832    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77d9cbf8b4-wmqzj" podStartSLOduration=1.53735742 podStartE2EDuration="4.530813005s" totalImagesPullingTime="2.993455585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:10.530808518 +0000 UTC m=+469.320965635" watchObservedRunningTime="2026-07-23 18:07:10.530813005 +0000 UTC m=+469.320970120"
Jul 23 18:07:12.522891 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:12.522750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl" event={"ID":"837026c8-64fa-4afa-b45c-3bbb0d092487","Type":"ContainerStarted","Data":"53d996bf89d6a325c106c4b495524b9f4a0e3ba092281c639a600b2cbc778ff3"}
Jul 23 18:07:12.523350 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:12.522986    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:12.547204 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:12.547148    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl" podStartSLOduration=2.7602312270000002 podStartE2EDuration="6.547129021s" totalImagesPullingTime="3.786897794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:12.546201888 +0000 UTC m=+471.336359003" watchObservedRunningTime="2026-07-23 18:07:12.547129021 +0000 UTC m=+471.337286136"
Jul 23 18:07:13.529234 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:13.529199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7" event={"ID":"a79a2916-00f0-47b2-b9a2-60bd57c5b596","Type":"ContainerStarted","Data":"a691ad1a1da4ef959c0bc35c4af50a9a14724021c28d90501daa689ba906f9eb"}
Jul 23 18:07:13.529713 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:13.529392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:13.589334 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:13.589267    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7" podStartSLOduration=2.066482247 podStartE2EDuration="5.589245785s" totalImagesPullingTime="3.522763538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:13.58472047 +0000 UTC m=+472.374877584" watchObservedRunningTime="2026-07-23 18:07:13.589245785 +0000 UTC m=+472.379402901"
Jul 23 18:07:13.943063 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:13.943024    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 18:07:13.943162 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:13.943092    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 18:07:14.535410 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:14.535369    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" event={"ID":"afeb4112-e048-45e4-b25f-369eae3d9c35","Type":"ContainerStarted","Data":"dec17de3fd6f84bdb2c738090f908129527613b426f282cd5b9e64fb0f90846d"}
Jul 23 18:07:14.594740 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:14.594679    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" podStartSLOduration=2.12876657 podStartE2EDuration="5.594658905s" totalImagesPullingTime="3.465892335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:14.594098291 +0000 UTC m=+473.384255407" watchObservedRunningTime="2026-07-23 18:07:14.594658905 +0000 UTC m=+473.384816042"
Jul 23 18:07:15.540008 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:15.539977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:15.541772 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:15.541744    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-5ch84 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 18:07:15.541884 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:15.541790    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" podUID="afeb4112-e048-45e4-b25f-369eae3d9c35" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 18:07:16.544235 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:16.544204    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:21.467261 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.467228    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pc744"]
Jul 23 18:07:21.470878 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.470863    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.503353 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:07:21.503310    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-controller-manager-dockercfg-9mxsk\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9mxsk\"" type="*v1.Secret"
Jul 23 18:07:21.503672 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:07:21.503647    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kserve-controller-manager-d6f4b65b8-pc744\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="cdff1e77-7007-4433-baa9-c4a3276613ad" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.503758 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:07:21.503678    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret"
Jul 23 18:07:21.572672 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.572628    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"]
Jul 23 18:07:21.577134 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.577116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:21.593545 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.593519    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pc744"]
Jul 23 18:07:21.607534 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.607499    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d4ll9\""
Jul 23 18:07:21.607811 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.607794    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 23 18:07:21.623601 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.623581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdff1e77-7007-4433-baa9-c4a3276613ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-pc744\" (UID: \"cdff1e77-7007-4433-baa9-c4a3276613ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.623697 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.623621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgqf\" (UniqueName: \"kubernetes.io/projected/cdff1e77-7007-4433-baa9-c4a3276613ad-kube-api-access-ndgqf\") pod \"kserve-controller-manager-d6f4b65b8-pc744\" (UID: \"cdff1e77-7007-4433-baa9-c4a3276613ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.665504 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.665471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"]
Jul 23 18:07:21.724715 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.724632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdff1e77-7007-4433-baa9-c4a3276613ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-pc744\" (UID: \"cdff1e77-7007-4433-baa9-c4a3276613ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.724715 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.724672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f610a2-b455-44a4-8a4e-229a376e4dcc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p5bpm\" (UID: \"48f610a2-b455-44a4-8a4e-229a376e4dcc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:21.724715 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.724696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgqf\" (UniqueName: \"kubernetes.io/projected/cdff1e77-7007-4433-baa9-c4a3276613ad-kube-api-access-ndgqf\") pod \"kserve-controller-manager-d6f4b65b8-pc744\" (UID: \"cdff1e77-7007-4433-baa9-c4a3276613ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.724715 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.724723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw4dm\" (UniqueName: \"kubernetes.io/projected/48f610a2-b455-44a4-8a4e-229a376e4dcc-kube-api-access-qw4dm\") pod \"llmisvc-controller-manager-777c6bd7d5-p5bpm\" (UID: \"48f610a2-b455-44a4-8a4e-229a376e4dcc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:21.822820 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.822785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgqf\" (UniqueName: \"kubernetes.io/projected/cdff1e77-7007-4433-baa9-c4a3276613ad-kube-api-access-ndgqf\") pod \"kserve-controller-manager-d6f4b65b8-pc744\" (UID: \"cdff1e77-7007-4433-baa9-c4a3276613ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:21.825670 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.825643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f610a2-b455-44a4-8a4e-229a376e4dcc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p5bpm\" (UID: \"48f610a2-b455-44a4-8a4e-229a376e4dcc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:21.825740 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.825696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw4dm\" (UniqueName: \"kubernetes.io/projected/48f610a2-b455-44a4-8a4e-229a376e4dcc-kube-api-access-qw4dm\") pod \"llmisvc-controller-manager-777c6bd7d5-p5bpm\" (UID: \"48f610a2-b455-44a4-8a4e-229a376e4dcc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:21.828371 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.828343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f610a2-b455-44a4-8a4e-229a376e4dcc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p5bpm\" (UID: \"48f610a2-b455-44a4-8a4e-229a376e4dcc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:21.975702 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:21.975635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw4dm\" (UniqueName: \"kubernetes.io/projected/48f610a2-b455-44a4-8a4e-229a376e4dcc-kube-api-access-qw4dm\") pod \"llmisvc-controller-manager-777c6bd7d5-p5bpm\" (UID: \"48f610a2-b455-44a4-8a4e-229a376e4dcc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:22.185991 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.185956    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:22.397874 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.397800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"]
Jul 23 18:07:22.400183 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:07:22.400159    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48f610a2_b455_44a4_8a4e_229a376e4dcc.slice/crio-4c1d50f940b1e727c2b60e01447badfec330b5015e72934bc9f36a58b9fa1692 WatchSource:0}: Error finding container 4c1d50f940b1e727c2b60e01447badfec330b5015e72934bc9f36a58b9fa1692: Status 404 returned error can't find the container with id 4c1d50f940b1e727c2b60e01447badfec330b5015e72934bc9f36a58b9fa1692
Jul 23 18:07:22.505918 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.505879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9mxsk\""
Jul 23 18:07:22.511277 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.511253    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 23 18:07:22.517389 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.517366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdff1e77-7007-4433-baa9-c4a3276613ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-pc744\" (UID: \"cdff1e77-7007-4433-baa9-c4a3276613ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:22.575332 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.575302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm" event={"ID":"48f610a2-b455-44a4-8a4e-229a376e4dcc","Type":"ContainerStarted","Data":"4c1d50f940b1e727c2b60e01447badfec330b5015e72934bc9f36a58b9fa1692"}
Jul 23 18:07:22.680594 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.680557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:22.831630 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:22.831599    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pc744"]
Jul 23 18:07:22.837575 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:07:22.837552    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdff1e77_7007_4433_baa9_c4a3276613ad.slice/crio-b77e1cc668618259e5e4a1471a0b697c6bcc5fb0d8414e9115cc26d62cc27d20 WatchSource:0}: Error finding container b77e1cc668618259e5e4a1471a0b697c6bcc5fb0d8414e9115cc26d62cc27d20: Status 404 returned error can't find the container with id b77e1cc668618259e5e4a1471a0b697c6bcc5fb0d8414e9115cc26d62cc27d20
Jul 23 18:07:23.532325 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:23.532292    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jg4bl"
Jul 23 18:07:23.583662 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:23.583630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744" event={"ID":"cdff1e77-7007-4433-baa9-c4a3276613ad","Type":"ContainerStarted","Data":"b77e1cc668618259e5e4a1471a0b697c6bcc5fb0d8414e9115cc26d62cc27d20"}
Jul 23 18:07:24.451458 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.451422    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"]
Jul 23 18:07:24.451688 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.451659    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" podUID="afeb4112-e048-45e4-b25f-369eae3d9c35" containerName="discovery" containerID="cri-o://dec17de3fd6f84bdb2c738090f908129527613b426f282cd5b9e64fb0f90846d" gracePeriod=30
Jul 23 18:07:24.539320 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.539295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-gngw7"
Jul 23 18:07:24.590932 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.590895    2562 generic.go:356] "Generic (PLEG): container finished" podID="afeb4112-e048-45e4-b25f-369eae3d9c35" containerID="dec17de3fd6f84bdb2c738090f908129527613b426f282cd5b9e64fb0f90846d" exitCode=0
Jul 23 18:07:24.591118 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.591007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" event={"ID":"afeb4112-e048-45e4-b25f-369eae3d9c35","Type":"ContainerDied","Data":"dec17de3fd6f84bdb2c738090f908129527613b426f282cd5b9e64fb0f90846d"}
Jul 23 18:07:24.722911 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.722881    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:24.852295 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852266    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-ca-configmap\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852306    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb4112-e048-45e4-b25f-369eae3d9c35-local-certs\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852328    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6zzw\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-kube-api-access-r6zzw\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-dns-cert\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852390    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-token\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852446    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-cacerts\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-kubeconfig\") pod \"afeb4112-e048-45e4-b25f-369eae3d9c35\" (UID: \"afeb4112-e048-45e4-b25f-369eae3d9c35\") "
Jul 23 18:07:24.852794 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.852759    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:07:24.855525 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.855486    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeb4112-e048-45e4-b25f-369eae3d9c35-local-certs" (OuterVolumeSpecName: "local-certs") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:07:24.855836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.855793    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-cacerts" (OuterVolumeSpecName: "cacerts") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:07:24.855969 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.855921    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:07:24.856043 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.856009    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-token" (OuterVolumeSpecName: "istio-token") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:07:24.856097 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.856057    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:07:24.856491 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.856467    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-kube-api-access-r6zzw" (OuterVolumeSpecName: "kube-api-access-r6zzw") pod "afeb4112-e048-45e4-b25f-369eae3d9c35" (UID: "afeb4112-e048-45e4-b25f-369eae3d9c35"). InnerVolumeSpecName "kube-api-access-r6zzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:07:24.953313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953233    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afeb4112-e048-45e4-b25f-369eae3d9c35-local-certs\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:24.953313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953268    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zzw\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-kube-api-access-r6zzw\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:24.953313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953283    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-dns-cert\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:24.953313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953296    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-token\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:24.953313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953308    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-cacerts\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:24.953313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953316    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-kubeconfig\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:24.953679 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:24.953324    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afeb4112-e048-45e4-b25f-369eae3d9c35-istio-csr-ca-configmap\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:07:25.598231 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:25.598195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84" event={"ID":"afeb4112-e048-45e4-b25f-369eae3d9c35","Type":"ContainerDied","Data":"f3f1073efe81d1691c6b85d8eec60300926e656b19e3f1bfd685e933f871fcd4"}
Jul 23 18:07:25.598712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:25.598245    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"
Jul 23 18:07:25.598712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:25.598266    2562 scope.go:117] "RemoveContainer" containerID="dec17de3fd6f84bdb2c738090f908129527613b426f282cd5b9e64fb0f90846d"
Jul 23 18:07:25.659878 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:25.659815    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"]
Jul 23 18:07:25.661810 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:25.661782    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5ch84"]
Jul 23 18:07:26.603541 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:26.603503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm" event={"ID":"48f610a2-b455-44a4-8a4e-229a376e4dcc","Type":"ContainerStarted","Data":"8c9c64b5483608f97ef80c42a4bd0a59ce8f745289719d1609dc9b5ff1c2bdd5"}
Jul 23 18:07:26.603985 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:26.603616    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:26.605886 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:26.605853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744" event={"ID":"cdff1e77-7007-4433-baa9-c4a3276613ad","Type":"ContainerStarted","Data":"3b5cb1bd028f06d4b31839ea7a27e0d7d70a80f7da9f2463deb591bac089c458"}
Jul 23 18:07:26.606020 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:26.605974    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:07:26.657508 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:26.657455    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm" podStartSLOduration=2.117674827 podStartE2EDuration="5.657437911s" totalImagesPullingTime="3.539763084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:26.657117047 +0000 UTC m=+485.447274182" watchObservedRunningTime="2026-07-23 18:07:26.657437911 +0000 UTC m=+485.447595082"
Jul 23 18:07:26.690378 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:26.690330    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744" podStartSLOduration=2.5919447719999997 podStartE2EDuration="5.690313608s" totalImagesPullingTime="3.098368836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:26.688854265 +0000 UTC m=+485.479011382" watchObservedRunningTime="2026-07-23 18:07:26.690313608 +0000 UTC m=+485.480470723"
Jul 23 18:07:27.607578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:27.607543    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afeb4112-e048-45e4-b25f-369eae3d9c35" path="/var/lib/kubelet/pods/afeb4112-e048-45e4-b25f-369eae3d9c35/volumes"
Jul 23 18:07:57.611425 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:57.611392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p5bpm"
Jul 23 18:07:57.614275 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:07:57.614252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pc744"
Jul 23 18:08:07.815980 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.815922    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xn9dr"]
Jul 23 18:08:07.816397 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.816231    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afeb4112-e048-45e4-b25f-369eae3d9c35" containerName="discovery"
Jul 23 18:08:07.816397 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.816242    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="afeb4112-e048-45e4-b25f-369eae3d9c35" containerName="discovery"
Jul 23 18:08:07.816397 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.816298    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="afeb4112-e048-45e4-b25f-369eae3d9c35" containerName="discovery"
Jul 23 18:08:07.829758 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.829728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:07.834303 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.834276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 18:08:07.834449 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.834352    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-p8drh\""
Jul 23 18:08:07.834449 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.834354    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 18:08:07.850868 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.850838    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xn9dr"]
Jul 23 18:08:07.891640 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.891609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhthg\" (UniqueName: \"kubernetes.io/projected/61d67676-11ab-4552-9cb1-22471259eaa2-kube-api-access-mhthg\") pod \"kuadrant-operator-catalog-xn9dr\" (UID: \"61d67676-11ab-4552-9cb1-22471259eaa2\") " pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:07.993119 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:07.993077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhthg\" (UniqueName: \"kubernetes.io/projected/61d67676-11ab-4552-9cb1-22471259eaa2-kube-api-access-mhthg\") pod \"kuadrant-operator-catalog-xn9dr\" (UID: \"61d67676-11ab-4552-9cb1-22471259eaa2\") " pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:08.007277 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.007251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhthg\" (UniqueName: \"kubernetes.io/projected/61d67676-11ab-4552-9cb1-22471259eaa2-kube-api-access-mhthg\") pod \"kuadrant-operator-catalog-xn9dr\" (UID: \"61d67676-11ab-4552-9cb1-22471259eaa2\") " pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:08.131458 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.131371    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xn9dr"]
Jul 23 18:08:08.131623 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.131611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:08.276254 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.276111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xn9dr"]
Jul 23 18:08:08.279038 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:08.279005    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d67676_11ab_4552_9cb1_22471259eaa2.slice/crio-88281dd7134f8ad9c747951e813c558e34e86e842b8032d5213d72d0fcd3fe34 WatchSource:0}: Error finding container 88281dd7134f8ad9c747951e813c558e34e86e842b8032d5213d72d0fcd3fe34: Status 404 returned error can't find the container with id 88281dd7134f8ad9c747951e813c558e34e86e842b8032d5213d72d0fcd3fe34
Jul 23 18:08:08.351028 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.350995    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2r2vh"]
Jul 23 18:08:08.355560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.355542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:08.378897 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.378867    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2r2vh"]
Jul 23 18:08:08.397179 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.397105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppg9t\" (UniqueName: \"kubernetes.io/projected/e7133930-cd65-48e5-a808-6d0cb0c09d3d-kube-api-access-ppg9t\") pod \"kuadrant-operator-catalog-2r2vh\" (UID: \"e7133930-cd65-48e5-a808-6d0cb0c09d3d\") " pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:08.497862 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.497819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppg9t\" (UniqueName: \"kubernetes.io/projected/e7133930-cd65-48e5-a808-6d0cb0c09d3d-kube-api-access-ppg9t\") pod \"kuadrant-operator-catalog-2r2vh\" (UID: \"e7133930-cd65-48e5-a808-6d0cb0c09d3d\") " pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:08.507255 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.507227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppg9t\" (UniqueName: \"kubernetes.io/projected/e7133930-cd65-48e5-a808-6d0cb0c09d3d-kube-api-access-ppg9t\") pod \"kuadrant-operator-catalog-2r2vh\" (UID: \"e7133930-cd65-48e5-a808-6d0cb0c09d3d\") " pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:08.664270 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.664177    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:08.783358 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.783330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr" event={"ID":"61d67676-11ab-4552-9cb1-22471259eaa2","Type":"ContainerStarted","Data":"88281dd7134f8ad9c747951e813c558e34e86e842b8032d5213d72d0fcd3fe34"}
Jul 23 18:08:08.800887 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:08.800864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2r2vh"]
Jul 23 18:08:08.802988 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:08.802961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7133930_cd65_48e5_a808_6d0cb0c09d3d.slice/crio-5eb9742a1a7a34fbcca9e7fb6a0eac5ff7cb1ee5d7e351797e09df44a500d40c WatchSource:0}: Error finding container 5eb9742a1a7a34fbcca9e7fb6a0eac5ff7cb1ee5d7e351797e09df44a500d40c: Status 404 returned error can't find the container with id 5eb9742a1a7a34fbcca9e7fb6a0eac5ff7cb1ee5d7e351797e09df44a500d40c
Jul 23 18:08:09.792236 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:09.792074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh" event={"ID":"e7133930-cd65-48e5-a808-6d0cb0c09d3d","Type":"ContainerStarted","Data":"5eb9742a1a7a34fbcca9e7fb6a0eac5ff7cb1ee5d7e351797e09df44a500d40c"}
Jul 23 18:08:11.801688 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:11.801652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr" event={"ID":"61d67676-11ab-4552-9cb1-22471259eaa2","Type":"ContainerStarted","Data":"0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d"}
Jul 23 18:08:11.802168 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:11.801684    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr" podUID="61d67676-11ab-4552-9cb1-22471259eaa2" containerName="registry-server" containerID="cri-o://0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d" gracePeriod=2
Jul 23 18:08:11.803195 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:11.803170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh" event={"ID":"e7133930-cd65-48e5-a808-6d0cb0c09d3d","Type":"ContainerStarted","Data":"ab096819ca214c43aa645c4007b8dd2928388df67c9ef96e196fc31837c90333"}
Jul 23 18:08:11.820461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:11.820415    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr" podStartSLOduration=2.245236054 podStartE2EDuration="4.82039987s" totalImagesPullingTime="2.575163816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:11.818863935 +0000 UTC m=+530.609021048" watchObservedRunningTime="2026-07-23 18:08:11.82039987 +0000 UTC m=+530.610557033"
Jul 23 18:08:11.837597 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:11.837546    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh" podStartSLOduration=1.80579147 podStartE2EDuration="3.837529087s" totalImagesPullingTime="2.031737617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:11.83592603 +0000 UTC m=+530.626083146" watchObservedRunningTime="2026-07-23 18:08:11.837529087 +0000 UTC m=+530.627686252"
Jul 23 18:08:12.040862 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.040837    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:12.130605 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.130503    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhthg\" (UniqueName: \"kubernetes.io/projected/61d67676-11ab-4552-9cb1-22471259eaa2-kube-api-access-mhthg\") pod \"61d67676-11ab-4552-9cb1-22471259eaa2\" (UID: \"61d67676-11ab-4552-9cb1-22471259eaa2\") "
Jul 23 18:08:12.132769 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.132738    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d67676-11ab-4552-9cb1-22471259eaa2-kube-api-access-mhthg" (OuterVolumeSpecName: "kube-api-access-mhthg") pod "61d67676-11ab-4552-9cb1-22471259eaa2" (UID: "61d67676-11ab-4552-9cb1-22471259eaa2"). InnerVolumeSpecName "kube-api-access-mhthg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:08:12.231554 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.231501    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhthg\" (UniqueName: \"kubernetes.io/projected/61d67676-11ab-4552-9cb1-22471259eaa2-kube-api-access-mhthg\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:12.813124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.813038    2562 generic.go:356] "Generic (PLEG): container finished" podID="61d67676-11ab-4552-9cb1-22471259eaa2" containerID="0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d" exitCode=0
Jul 23 18:08:12.813124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.813097    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr"
Jul 23 18:08:12.813587 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.813127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr" event={"ID":"61d67676-11ab-4552-9cb1-22471259eaa2","Type":"ContainerDied","Data":"0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d"}
Jul 23 18:08:12.813587 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.813196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xn9dr" event={"ID":"61d67676-11ab-4552-9cb1-22471259eaa2","Type":"ContainerDied","Data":"88281dd7134f8ad9c747951e813c558e34e86e842b8032d5213d72d0fcd3fe34"}
Jul 23 18:08:12.813587 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.813217    2562 scope.go:117] "RemoveContainer" containerID="0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d"
Jul 23 18:08:12.822397 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.822377    2562 scope.go:117] "RemoveContainer" containerID="0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d"
Jul 23 18:08:12.822705 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:12.822674    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d\": container with ID starting with 0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d not found: ID does not exist" containerID="0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d"
Jul 23 18:08:12.822744 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.822714    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d"} err="failed to get container status \"0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d\": rpc error: code = NotFound desc = could not find container \"0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d\": container with ID starting with 0a1eb79373612471573f1b2ec13f3074206058f6a6fd31b714404875d017567d not found: ID does not exist"
Jul 23 18:08:12.835526 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.835498    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xn9dr"]
Jul 23 18:08:12.839999 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:12.839975    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xn9dr"]
Jul 23 18:08:13.606867 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:13.606834    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61d67676-11ab-4552-9cb1-22471259eaa2" path="/var/lib/kubelet/pods/61d67676-11ab-4552-9cb1-22471259eaa2/volumes"
Jul 23 18:08:18.664799 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:18.664751    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:18.665325 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:18.664829    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:18.687408 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:18.687377    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:18.862252 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:18.862227    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2r2vh"
Jul 23 18:08:20.839962 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.839907    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"]
Jul 23 18:08:20.840341 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.840234    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d67676-11ab-4552-9cb1-22471259eaa2" containerName="registry-server"
Jul 23 18:08:20.840341 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.840245    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d67676-11ab-4552-9cb1-22471259eaa2" containerName="registry-server"
Jul 23 18:08:20.840341 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.840316    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d67676-11ab-4552-9cb1-22471259eaa2" containerName="registry-server"
Jul 23 18:08:20.843376 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.843360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:20.850826 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.850552    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lrl9q\""
Jul 23 18:08:20.859306 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:20.859277    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"]
Jul 23 18:08:21.001933 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.001897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.001933 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.001933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.002161 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.001997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmvn\" (UniqueName: \"kubernetes.io/projected/ccba60c3-b4bf-40ca-bf09-16cf120e2546-kube-api-access-6tmvn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.103240 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.103153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.103240 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.103182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.103240 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.103201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmvn\" (UniqueName: \"kubernetes.io/projected/ccba60c3-b4bf-40ca-bf09-16cf120e2546-kube-api-access-6tmvn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.103539 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.103487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.103597 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.103563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.111728 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.111696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmvn\" (UniqueName: \"kubernetes.io/projected/ccba60c3-b4bf-40ca-bf09-16cf120e2546-kube-api-access-6tmvn\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.153347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.153321    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:21.300000 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.299978    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"]
Jul 23 18:08:21.302487 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:21.302449    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccba60c3_b4bf_40ca_bf09_16cf120e2546.slice/crio-22ac904811eb13d045b6e1d7f22d1a62ae4c698ac9b49b907679a7881154537a WatchSource:0}: Error finding container 22ac904811eb13d045b6e1d7f22d1a62ae4c698ac9b49b907679a7881154537a: Status 404 returned error can't find the container with id 22ac904811eb13d045b6e1d7f22d1a62ae4c698ac9b49b907679a7881154537a
Jul 23 18:08:21.455602 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.455566    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"]
Jul 23 18:08:21.459037 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.459021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.471633 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.471607    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"]
Jul 23 18:08:21.606281 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.606253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.606451 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.606432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchgt\" (UniqueName: \"kubernetes.io/projected/862e57b5-0b63-405f-bee2-529f0b760fcd-kube-api-access-jchgt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.606545 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.606531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.707645 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.707565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.707645 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.707618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.707866 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.707655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchgt\" (UniqueName: \"kubernetes.io/projected/862e57b5-0b63-405f-bee2-529f0b760fcd-kube-api-access-jchgt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.708016 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.707912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.708016 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.707966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.716418 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.716400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchgt\" (UniqueName: \"kubernetes.io/projected/862e57b5-0b63-405f-bee2-529f0b760fcd-kube-api-access-jchgt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.767337 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.767309    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:21.857599 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.857565    2562 generic.go:356] "Generic (PLEG): container finished" podID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerID="de2180f082ee8db606a8aab4dd024efdf7ccee48c5e7bfd1a86a89cf6270c5e0" exitCode=0
Jul 23 18:08:21.858068 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.857652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5" event={"ID":"ccba60c3-b4bf-40ca-bf09-16cf120e2546","Type":"ContainerDied","Data":"de2180f082ee8db606a8aab4dd024efdf7ccee48c5e7bfd1a86a89cf6270c5e0"}
Jul 23 18:08:21.858068 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.857711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5" event={"ID":"ccba60c3-b4bf-40ca-bf09-16cf120e2546","Type":"ContainerStarted","Data":"22ac904811eb13d045b6e1d7f22d1a62ae4c698ac9b49b907679a7881154537a"}
Jul 23 18:08:21.913420 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:21.913398    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"]
Jul 23 18:08:21.915894 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:21.915868    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862e57b5_0b63_405f_bee2_529f0b760fcd.slice/crio-d21cc8787b859861a421e015ef9c473d29968888c1023627e30e16896517ea2a WatchSource:0}: Error finding container d21cc8787b859861a421e015ef9c473d29968888c1023627e30e16896517ea2a: Status 404 returned error can't find the container with id d21cc8787b859861a421e015ef9c473d29968888c1023627e30e16896517ea2a
Jul 23 18:08:22.204135 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.204098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"]
Jul 23 18:08:22.207470 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.207453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.231080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.231054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"]
Jul 23 18:08:22.313614 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.313576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6qt\" (UniqueName: \"kubernetes.io/projected/0a995e83-dee5-40be-a698-016d9d77087b-kube-api-access-9h6qt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.313811 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.313640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.313811 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.313715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.414606 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.414573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.414773 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.414618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.414773 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.414669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6qt\" (UniqueName: \"kubernetes.io/projected/0a995e83-dee5-40be-a698-016d9d77087b-kube-api-access-9h6qt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.415009 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.414988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.415082 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.415031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.429916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.429884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6qt\" (UniqueName: \"kubernetes.io/projected/0a995e83-dee5-40be-a698-016d9d77087b-kube-api-access-9h6qt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.515390 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.515361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:22.680205 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.680185    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"]
Jul 23 18:08:22.834993 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.834964    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"]
Jul 23 18:08:22.838273 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.838255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:22.852875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.852848    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"]
Jul 23 18:08:22.864161 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.864136    2562 generic.go:356] "Generic (PLEG): container finished" podID="0a995e83-dee5-40be-a698-016d9d77087b" containerID="c265a849d11f1aff0c138ef7e24af9d2dfd23df0e8ad3bb779de06a6840152db" exitCode=0
Jul 23 18:08:22.864554 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.864216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6" event={"ID":"0a995e83-dee5-40be-a698-016d9d77087b","Type":"ContainerDied","Data":"c265a849d11f1aff0c138ef7e24af9d2dfd23df0e8ad3bb779de06a6840152db"}
Jul 23 18:08:22.864554 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.864260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6" event={"ID":"0a995e83-dee5-40be-a698-016d9d77087b","Type":"ContainerStarted","Data":"48e7cb6a4e38f92fd769c932dbc455922a064be858b2c900f08bad37b3d66adc"}
Jul 23 18:08:22.866038 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.866018    2562 generic.go:356] "Generic (PLEG): container finished" podID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerID="dfbcc16286cf078d60c0092a73700c99f6c76b79b0f3fe974683293e4d801575" exitCode=0
Jul 23 18:08:22.866137 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.866096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5" event={"ID":"ccba60c3-b4bf-40ca-bf09-16cf120e2546","Type":"ContainerDied","Data":"dfbcc16286cf078d60c0092a73700c99f6c76b79b0f3fe974683293e4d801575"}
Jul 23 18:08:22.867639 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.867621    2562 generic.go:356] "Generic (PLEG): container finished" podID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerID="eeddb5a1d47d450223cd4aa005ad2c222d74f666de4a5d82728b19e7bb057016" exitCode=0
Jul 23 18:08:22.867760 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.867664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj" event={"ID":"862e57b5-0b63-405f-bee2-529f0b760fcd","Type":"ContainerDied","Data":"eeddb5a1d47d450223cd4aa005ad2c222d74f666de4a5d82728b19e7bb057016"}
Jul 23 18:08:22.867760 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.867697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj" event={"ID":"862e57b5-0b63-405f-bee2-529f0b760fcd","Type":"ContainerStarted","Data":"d21cc8787b859861a421e015ef9c473d29968888c1023627e30e16896517ea2a"}
Jul 23 18:08:22.917225 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.917198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:22.917352 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.917317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xp4\" (UniqueName: \"kubernetes.io/projected/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-kube-api-access-b5xp4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:22.917352 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:22.917346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.018330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.018240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xp4\" (UniqueName: \"kubernetes.io/projected/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-kube-api-access-b5xp4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.018330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.018281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.018687 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.018343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.018791 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.018763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.018917 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.018896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.028542 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.028520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xp4\" (UniqueName: \"kubernetes.io/projected/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-kube-api-access-b5xp4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.186059 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.186024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:23.325579 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.325502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"]
Jul 23 18:08:23.328841 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:23.328804    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d28d85_751e_4db0_a300_dd4a1f6d2ac1.slice/crio-7f851cfeea2c231a471160cf12f047d3796e52074f5a3a13b21b46b4e3f7737a WatchSource:0}: Error finding container 7f851cfeea2c231a471160cf12f047d3796e52074f5a3a13b21b46b4e3f7737a: Status 404 returned error can't find the container with id 7f851cfeea2c231a471160cf12f047d3796e52074f5a3a13b21b46b4e3f7737a
Jul 23 18:08:23.874247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.874205    2562 generic.go:356] "Generic (PLEG): container finished" podID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerID="748690c869b750d50c3641ffcd951f518245b4f3fb3d9ff13c0fab97fcff3d67" exitCode=0
Jul 23 18:08:23.874655 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.874286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h" event={"ID":"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1","Type":"ContainerDied","Data":"748690c869b750d50c3641ffcd951f518245b4f3fb3d9ff13c0fab97fcff3d67"}
Jul 23 18:08:23.874655 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.874336    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h" event={"ID":"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1","Type":"ContainerStarted","Data":"7f851cfeea2c231a471160cf12f047d3796e52074f5a3a13b21b46b4e3f7737a"}
Jul 23 18:08:23.876688 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.876663    2562 generic.go:356] "Generic (PLEG): container finished" podID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerID="4ff606f088d7c34dca6ea5b3b10c66b7f737ebd78c2cdca32d95d413b81d6b4c" exitCode=0
Jul 23 18:08:23.876796 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.876748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5" event={"ID":"ccba60c3-b4bf-40ca-bf09-16cf120e2546","Type":"ContainerDied","Data":"4ff606f088d7c34dca6ea5b3b10c66b7f737ebd78c2cdca32d95d413b81d6b4c"}
Jul 23 18:08:23.878455 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.878429    2562 generic.go:356] "Generic (PLEG): container finished" podID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerID="c6967ac80ac6807c4bcd2e69656f6fa01a83f9dbf2f99766a7ad22d81b33fc0b" exitCode=0
Jul 23 18:08:23.878562 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.878461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj" event={"ID":"862e57b5-0b63-405f-bee2-529f0b760fcd","Type":"ContainerDied","Data":"c6967ac80ac6807c4bcd2e69656f6fa01a83f9dbf2f99766a7ad22d81b33fc0b"}
Jul 23 18:08:23.880564 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.880537    2562 generic.go:356] "Generic (PLEG): container finished" podID="0a995e83-dee5-40be-a698-016d9d77087b" containerID="a890b7251422c14f7536ac325842205d15563cce82ff5041846dcd71dfe3c54b" exitCode=0
Jul 23 18:08:23.880662 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:23.880564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6" event={"ID":"0a995e83-dee5-40be-a698-016d9d77087b","Type":"ContainerDied","Data":"a890b7251422c14f7536ac325842205d15563cce82ff5041846dcd71dfe3c54b"}
Jul 23 18:08:24.887989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:24.887881    2562 generic.go:356] "Generic (PLEG): container finished" podID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerID="42ce3d37512570d3dd60e3e748622fb3db8173c93aef57636ac826dbf81cef38" exitCode=0
Jul 23 18:08:24.887989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:24.887960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj" event={"ID":"862e57b5-0b63-405f-bee2-529f0b760fcd","Type":"ContainerDied","Data":"42ce3d37512570d3dd60e3e748622fb3db8173c93aef57636ac826dbf81cef38"}
Jul 23 18:08:24.890009 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:24.889982    2562 generic.go:356] "Generic (PLEG): container finished" podID="0a995e83-dee5-40be-a698-016d9d77087b" containerID="b77f610ce1d3fe62ad2ecb520ffdc101092af25f741b0db49d7bf09f3fbf56e9" exitCode=0
Jul 23 18:08:24.890111 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:24.890066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6" event={"ID":"0a995e83-dee5-40be-a698-016d9d77087b","Type":"ContainerDied","Data":"b77f610ce1d3fe62ad2ecb520ffdc101092af25f741b0db49d7bf09f3fbf56e9"}
Jul 23 18:08:24.891678 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:24.891660    2562 generic.go:356] "Generic (PLEG): container finished" podID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerID="fec71f3a65ec8f125cb93c6d182f173f014b5a241c64e018c66411e67c6f8cbb" exitCode=0
Jul 23 18:08:24.891773 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:24.891728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h" event={"ID":"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1","Type":"ContainerDied","Data":"fec71f3a65ec8f125cb93c6d182f173f014b5a241c64e018c66411e67c6f8cbb"}
Jul 23 18:08:25.022885 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.022863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:25.137034 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.136998    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-util\") pod \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") "
Jul 23 18:08:25.137216 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.137065    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tmvn\" (UniqueName: \"kubernetes.io/projected/ccba60c3-b4bf-40ca-bf09-16cf120e2546-kube-api-access-6tmvn\") pod \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") "
Jul 23 18:08:25.137216 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.137093    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-bundle\") pod \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\" (UID: \"ccba60c3-b4bf-40ca-bf09-16cf120e2546\") "
Jul 23 18:08:25.137766 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.137735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-bundle" (OuterVolumeSpecName: "bundle") pod "ccba60c3-b4bf-40ca-bf09-16cf120e2546" (UID: "ccba60c3-b4bf-40ca-bf09-16cf120e2546"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:25.139304 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.139253    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccba60c3-b4bf-40ca-bf09-16cf120e2546-kube-api-access-6tmvn" (OuterVolumeSpecName: "kube-api-access-6tmvn") pod "ccba60c3-b4bf-40ca-bf09-16cf120e2546" (UID: "ccba60c3-b4bf-40ca-bf09-16cf120e2546"). InnerVolumeSpecName "kube-api-access-6tmvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:08:25.142231 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.142196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-util" (OuterVolumeSpecName: "util") pod "ccba60c3-b4bf-40ca-bf09-16cf120e2546" (UID: "ccba60c3-b4bf-40ca-bf09-16cf120e2546"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:25.238030 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.237972    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmvn\" (UniqueName: \"kubernetes.io/projected/ccba60c3-b4bf-40ca-bf09-16cf120e2546-kube-api-access-6tmvn\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:25.238030 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.238018    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:25.238030 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.238028    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccba60c3-b4bf-40ca-bf09-16cf120e2546-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:25.898502 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.898409    2562 generic.go:356] "Generic (PLEG): container finished" podID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerID="23cd02f9b52e790348bfc6ea794fe00824c88b7e0f400e3cf066d4c52648fa50" exitCode=0
Jul 23 18:08:25.898913 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.898509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h" event={"ID":"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1","Type":"ContainerDied","Data":"23cd02f9b52e790348bfc6ea794fe00824c88b7e0f400e3cf066d4c52648fa50"}
Jul 23 18:08:25.900401 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.900382    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5"
Jul 23 18:08:25.900513 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.900414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5" event={"ID":"ccba60c3-b4bf-40ca-bf09-16cf120e2546","Type":"ContainerDied","Data":"22ac904811eb13d045b6e1d7f22d1a62ae4c698ac9b49b907679a7881154537a"}
Jul 23 18:08:25.900513 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:25.900453    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ac904811eb13d045b6e1d7f22d1a62ae4c698ac9b49b907679a7881154537a"
Jul 23 18:08:26.025912 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.025887    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:26.054488 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.054461    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:26.146716 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.146685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h6qt\" (UniqueName: \"kubernetes.io/projected/0a995e83-dee5-40be-a698-016d9d77087b-kube-api-access-9h6qt\") pod \"0a995e83-dee5-40be-a698-016d9d77087b\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") "
Jul 23 18:08:26.146716 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.146722    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-bundle\") pod \"0a995e83-dee5-40be-a698-016d9d77087b\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") "
Jul 23 18:08:26.146984 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.146781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jchgt\" (UniqueName: \"kubernetes.io/projected/862e57b5-0b63-405f-bee2-529f0b760fcd-kube-api-access-jchgt\") pod \"862e57b5-0b63-405f-bee2-529f0b760fcd\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") "
Jul 23 18:08:26.146984 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.146811    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-util\") pod \"0a995e83-dee5-40be-a698-016d9d77087b\" (UID: \"0a995e83-dee5-40be-a698-016d9d77087b\") "
Jul 23 18:08:26.146984 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.146832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-bundle\") pod \"862e57b5-0b63-405f-bee2-529f0b760fcd\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") "
Jul 23 18:08:26.146984 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.146846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-util\") pod \"862e57b5-0b63-405f-bee2-529f0b760fcd\" (UID: \"862e57b5-0b63-405f-bee2-529f0b760fcd\") "
Jul 23 18:08:26.147441 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.147408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-bundle" (OuterVolumeSpecName: "bundle") pod "0a995e83-dee5-40be-a698-016d9d77087b" (UID: "0a995e83-dee5-40be-a698-016d9d77087b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:26.147573 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.147481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-bundle" (OuterVolumeSpecName: "bundle") pod "862e57b5-0b63-405f-bee2-529f0b760fcd" (UID: "862e57b5-0b63-405f-bee2-529f0b760fcd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:26.149160 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.149110    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a995e83-dee5-40be-a698-016d9d77087b-kube-api-access-9h6qt" (OuterVolumeSpecName: "kube-api-access-9h6qt") pod "0a995e83-dee5-40be-a698-016d9d77087b" (UID: "0a995e83-dee5-40be-a698-016d9d77087b"). InnerVolumeSpecName "kube-api-access-9h6qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:08:26.149160 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.149143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862e57b5-0b63-405f-bee2-529f0b760fcd-kube-api-access-jchgt" (OuterVolumeSpecName: "kube-api-access-jchgt") pod "862e57b5-0b63-405f-bee2-529f0b760fcd" (UID: "862e57b5-0b63-405f-bee2-529f0b760fcd"). InnerVolumeSpecName "kube-api-access-jchgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:08:26.152046 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.152015    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-util" (OuterVolumeSpecName: "util") pod "0a995e83-dee5-40be-a698-016d9d77087b" (UID: "0a995e83-dee5-40be-a698-016d9d77087b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:26.152317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.152298    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-util" (OuterVolumeSpecName: "util") pod "862e57b5-0b63-405f-bee2-529f0b760fcd" (UID: "862e57b5-0b63-405f-bee2-529f0b760fcd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:26.248366 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.248332    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:26.248366 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.248362    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jchgt\" (UniqueName: \"kubernetes.io/projected/862e57b5-0b63-405f-bee2-529f0b760fcd-kube-api-access-jchgt\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:26.248366 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.248372    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a995e83-dee5-40be-a698-016d9d77087b-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:26.248587 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.248380    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:26.248587 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.248388    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/862e57b5-0b63-405f-bee2-529f0b760fcd-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:26.248587 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.248395    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h6qt\" (UniqueName: \"kubernetes.io/projected/0a995e83-dee5-40be-a698-016d9d77087b-kube-api-access-9h6qt\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:26.907907 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.907873    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj"
Jul 23 18:08:26.907907 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.907895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj" event={"ID":"862e57b5-0b63-405f-bee2-529f0b760fcd","Type":"ContainerDied","Data":"d21cc8787b859861a421e015ef9c473d29968888c1023627e30e16896517ea2a"}
Jul 23 18:08:26.908380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.907964    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d21cc8787b859861a421e015ef9c473d29968888c1023627e30e16896517ea2a"
Jul 23 18:08:26.910913 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.910865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6" event={"ID":"0a995e83-dee5-40be-a698-016d9d77087b","Type":"ContainerDied","Data":"48e7cb6a4e38f92fd769c932dbc455922a064be858b2c900f08bad37b3d66adc"}
Jul 23 18:08:26.911550 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.910959    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6"
Jul 23 18:08:26.911673 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:26.911656    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e7cb6a4e38f92fd769c932dbc455922a064be858b2c900f08bad37b3d66adc"
Jul 23 18:08:27.056205 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.056181    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:27.155170 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.155137    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-util\") pod \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") "
Jul 23 18:08:27.155333 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.155189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-bundle\") pod \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") "
Jul 23 18:08:27.155333 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.155256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5xp4\" (UniqueName: \"kubernetes.io/projected/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-kube-api-access-b5xp4\") pod \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\" (UID: \"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1\") "
Jul 23 18:08:27.155671 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.155650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-bundle" (OuterVolumeSpecName: "bundle") pod "c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" (UID: "c5d28d85-751e-4db0-a300-dd4a1f6d2ac1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:27.157567 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.157533    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-kube-api-access-b5xp4" (OuterVolumeSpecName: "kube-api-access-b5xp4") pod "c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" (UID: "c5d28d85-751e-4db0-a300-dd4a1f6d2ac1"). InnerVolumeSpecName "kube-api-access-b5xp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:08:27.160887 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.160831    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-util" (OuterVolumeSpecName: "util") pod "c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" (UID: "c5d28d85-751e-4db0-a300-dd4a1f6d2ac1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:08:27.256543 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.256500    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xp4\" (UniqueName: \"kubernetes.io/projected/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-kube-api-access-b5xp4\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:27.256543 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.256536    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-util\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:27.256543 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.256549    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d28d85-751e-4db0-a300-dd4a1f6d2ac1-bundle\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:08:27.916727 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.916645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h" event={"ID":"c5d28d85-751e-4db0-a300-dd4a1f6d2ac1","Type":"ContainerDied","Data":"7f851cfeea2c231a471160cf12f047d3796e52074f5a3a13b21b46b4e3f7737a"}
Jul 23 18:08:27.916727 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.916670    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h"
Jul 23 18:08:27.916727 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:27.916692    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f851cfeea2c231a471160cf12f047d3796e52074f5a3a13b21b46b4e3f7737a"
Jul 23 18:08:34.810441 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810403    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"]
Jul 23 18:08:34.810836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810821    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="extract"
Jul 23 18:08:34.810875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810840    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="extract"
Jul 23 18:08:34.810875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810853    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="util"
Jul 23 18:08:34.810875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810860    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="util"
Jul 23 18:08:34.810875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810873    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="util"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810880    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="util"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810893    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810900    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810909    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810916    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810927    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810934    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810959    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="extract"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810967    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="extract"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810980    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="util"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810987    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="util"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.810997    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="extract"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811004    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="extract"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811015    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="util"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811022    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="util"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811037    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="extract"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811045    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="extract"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811056    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="pull"
Jul 23 18:08:34.811080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811063    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="pull"
Jul 23 18:08:34.811836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811139    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccba60c3-b4bf-40ca-bf09-16cf120e2546" containerName="extract"
Jul 23 18:08:34.811836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="862e57b5-0b63-405f-bee2-529f0b760fcd" containerName="extract"
Jul 23 18:08:34.811836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811158    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a995e83-dee5-40be-a698-016d9d77087b" containerName="extract"
Jul 23 18:08:34.811836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.811166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d28d85-751e-4db0-a300-dd4a1f6d2ac1" containerName="extract"
Jul 23 18:08:34.814628 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.814605    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:34.817347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.817324    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ppxlz\""
Jul 23 18:08:34.837327 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.837300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"]
Jul 23 18:08:34.915685 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.915649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/513e3773-e800-4a26-bc48-62e234de0f94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:34.915852 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:34.915707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqvw\" (UniqueName: \"kubernetes.io/projected/513e3773-e800-4a26-bc48-62e234de0f94-kube-api-access-bqqvw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:35.017035 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.016984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqvw\" (UniqueName: \"kubernetes.io/projected/513e3773-e800-4a26-bc48-62e234de0f94-kube-api-access-bqqvw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:35.017216 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.017096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/513e3773-e800-4a26-bc48-62e234de0f94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:35.017430 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.017413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/513e3773-e800-4a26-bc48-62e234de0f94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:35.027264 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.027242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqvw\" (UniqueName: \"kubernetes.io/projected/513e3773-e800-4a26-bc48-62e234de0f94-kube-api-access-bqqvw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:35.124524 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.124438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:35.270037 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.269927    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"]
Jul 23 18:08:35.272891 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:35.272864    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513e3773_e800_4a26_bc48_62e234de0f94.slice/crio-7340c52cf27d0a631ee214cce671bad6d1609b9ad6ed5a4725ff28f1fc42c376 WatchSource:0}: Error finding container 7340c52cf27d0a631ee214cce671bad6d1609b9ad6ed5a4725ff28f1fc42c376: Status 404 returned error can't find the container with id 7340c52cf27d0a631ee214cce671bad6d1609b9ad6ed5a4725ff28f1fc42c376
Jul 23 18:08:35.955603 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:35.955567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87" event={"ID":"513e3773-e800-4a26-bc48-62e234de0f94","Type":"ContainerStarted","Data":"7340c52cf27d0a631ee214cce671bad6d1609b9ad6ed5a4725ff28f1fc42c376"}
Jul 23 18:08:36.663041 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.663001    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"]
Jul 23 18:08:36.666266 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.666250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:36.669453 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.669434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ts7pw\""
Jul 23 18:08:36.670075 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.670058    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 23 18:08:36.717379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.717350    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"]
Jul 23 18:08:36.836556 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.836523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhx2p\" (UniqueName: \"kubernetes.io/projected/1133f77b-042a-470c-9942-b913b9451c0f-kube-api-access-vhx2p\") pod \"dns-operator-controller-manager-864dd78c4c-s77t9\" (UID: \"1133f77b-042a-470c-9942-b913b9451c0f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:36.937318 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.937240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhx2p\" (UniqueName: \"kubernetes.io/projected/1133f77b-042a-470c-9942-b913b9451c0f-kube-api-access-vhx2p\") pod \"dns-operator-controller-manager-864dd78c4c-s77t9\" (UID: \"1133f77b-042a-470c-9942-b913b9451c0f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:36.964592 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.964562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhx2p\" (UniqueName: \"kubernetes.io/projected/1133f77b-042a-470c-9942-b913b9451c0f-kube-api-access-vhx2p\") pod \"dns-operator-controller-manager-864dd78c4c-s77t9\" (UID: \"1133f77b-042a-470c-9942-b913b9451c0f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:36.976378 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:36.976350    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:37.146966 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:37.146912    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"]
Jul 23 18:08:37.149797 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:37.149763    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1133f77b_042a_470c_9942_b913b9451c0f.slice/crio-8310ac4c26c7e28c78977581fd763e865591db1857273224849369dfb390deb1 WatchSource:0}: Error finding container 8310ac4c26c7e28c78977581fd763e865591db1857273224849369dfb390deb1: Status 404 returned error can't find the container with id 8310ac4c26c7e28c78977581fd763e865591db1857273224849369dfb390deb1
Jul 23 18:08:37.969104 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:37.969063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9" event={"ID":"1133f77b-042a-470c-9942-b913b9451c0f","Type":"ContainerStarted","Data":"8310ac4c26c7e28c78977581fd763e865591db1857273224849369dfb390deb1"}
Jul 23 18:08:39.983005 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:39.982970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9" event={"ID":"1133f77b-042a-470c-9942-b913b9451c0f","Type":"ContainerStarted","Data":"22581e6e78e8b19e32a34dc7af65d5036c44a639d2cd2a48f77db3947bbc0d03"}
Jul 23 18:08:39.983454 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:39.983112    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:40.007180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.007121    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9" podStartSLOduration=1.9621347230000001 podStartE2EDuration="4.007100099s" totalImagesPullingTime="2.044965376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:40.006199217 +0000 UTC m=+558.796356355" watchObservedRunningTime="2026-07-23 18:08:40.007100099 +0000 UTC m=+558.797257215"
Jul 23 18:08:40.339781 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.339746    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"]
Jul 23 18:08:40.343572 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.343551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:40.351396 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.351346    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pb5np\""
Jul 23 18:08:40.367274 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.367244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65k8j\" (UniqueName: \"kubernetes.io/projected/a65123fc-7d50-4a0c-9fab-f291760dac5b-kube-api-access-65k8j\") pod \"authorino-operator-75cc8d5cb7-kh4pr\" (UID: \"a65123fc-7d50-4a0c-9fab-f291760dac5b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:40.374697 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.374674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"]
Jul 23 18:08:40.468247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.468202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65k8j\" (UniqueName: \"kubernetes.io/projected/a65123fc-7d50-4a0c-9fab-f291760dac5b-kube-api-access-65k8j\") pod \"authorino-operator-75cc8d5cb7-kh4pr\" (UID: \"a65123fc-7d50-4a0c-9fab-f291760dac5b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:40.497323 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.497285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65k8j\" (UniqueName: \"kubernetes.io/projected/a65123fc-7d50-4a0c-9fab-f291760dac5b-kube-api-access-65k8j\") pod \"authorino-operator-75cc8d5cb7-kh4pr\" (UID: \"a65123fc-7d50-4a0c-9fab-f291760dac5b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:40.655621 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.655535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:40.990764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:40.990730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"]
Jul 23 18:08:40.991916 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:40.991894    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65123fc_7d50_4a0c_9fab_f291760dac5b.slice/crio-bc5756f9057870e3023da411debfee37470bc15bd1a1247aa89e85ff7756530d WatchSource:0}: Error finding container bc5756f9057870e3023da411debfee37470bc15bd1a1247aa89e85ff7756530d: Status 404 returned error can't find the container with id bc5756f9057870e3023da411debfee37470bc15bd1a1247aa89e85ff7756530d
Jul 23 18:08:41.999186 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:41.999144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr" event={"ID":"a65123fc-7d50-4a0c-9fab-f291760dac5b","Type":"ContainerStarted","Data":"bc5756f9057870e3023da411debfee37470bc15bd1a1247aa89e85ff7756530d"}
Jul 23 18:08:42.001303 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:42.001264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87" event={"ID":"513e3773-e800-4a26-bc48-62e234de0f94","Type":"ContainerStarted","Data":"da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba"}
Jul 23 18:08:42.001542 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:42.001425    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:42.039349 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:42.039283    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87" podStartSLOduration=2.407054127 podStartE2EDuration="8.039257183s" totalImagesPullingTime="5.632203056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:42.035241863 +0000 UTC m=+560.825398982" watchObservedRunningTime="2026-07-23 18:08:42.039257183 +0000 UTC m=+560.829414298"
Jul 23 18:08:43.016073 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.016039    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"]
Jul 23 18:08:43.019319 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.019302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:43.022236 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.022207    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-df459\""
Jul 23 18:08:43.031996 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.031968    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"]
Jul 23 18:08:43.086574 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.086536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5vs\" (UniqueName: \"kubernetes.io/projected/8b1c240f-ee89-49cb-b380-7b921dc62bda-kube-api-access-kp5vs\") pod \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" (UID: \"8b1c240f-ee89-49cb-b380-7b921dc62bda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:43.187780 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.187736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5vs\" (UniqueName: \"kubernetes.io/projected/8b1c240f-ee89-49cb-b380-7b921dc62bda-kube-api-access-kp5vs\") pod \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" (UID: \"8b1c240f-ee89-49cb-b380-7b921dc62bda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:43.198103 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.198069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5vs\" (UniqueName: \"kubernetes.io/projected/8b1c240f-ee89-49cb-b380-7b921dc62bda-kube-api-access-kp5vs\") pod \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" (UID: \"8b1c240f-ee89-49cb-b380-7b921dc62bda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:43.329782 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.329583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:43.472591 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:43.472567    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"]
Jul 23 18:08:43.474434 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:43.474407    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1c240f_ee89_49cb_b380_7b921dc62bda.slice/crio-7ebfaa8a4d09f0412bd33a717d149b7e3bbd8be793b6db00b82486d68205cfa6 WatchSource:0}: Error finding container 7ebfaa8a4d09f0412bd33a717d149b7e3bbd8be793b6db00b82486d68205cfa6: Status 404 returned error can't find the container with id 7ebfaa8a4d09f0412bd33a717d149b7e3bbd8be793b6db00b82486d68205cfa6
Jul 23 18:08:44.012722 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.012688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt" event={"ID":"8b1c240f-ee89-49cb-b380-7b921dc62bda","Type":"ContainerStarted","Data":"7ebfaa8a4d09f0412bd33a717d149b7e3bbd8be793b6db00b82486d68205cfa6"}
Jul 23 18:08:44.017485 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.014401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr" event={"ID":"a65123fc-7d50-4a0c-9fab-f291760dac5b","Type":"ContainerStarted","Data":"14a58180f062f3c7ca3b9234cb8497d9418597f310d6135e2321bf4b1a229705"}
Jul 23 18:08:44.017485 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.014679    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:44.096177 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.096124    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr" podStartSLOduration=1.7592942040000001 podStartE2EDuration="4.096105313s" totalImagesPullingTime="2.336811109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:44.045330935 +0000 UTC m=+562.835488049" watchObservedRunningTime="2026-07-23 18:08:44.096105313 +0000 UTC m=+562.886262428"
Jul 23 18:08:44.097878 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.097860    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"]
Jul 23 18:08:44.101863 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.101839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.104400 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.104373    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lrl9q\""
Jul 23 18:08:44.104879 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.104860    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 18:08:44.105014 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.104866    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 18:08:44.117133 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.116675    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"]
Jul 23 18:08:44.196739 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.196701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76ld\" (UniqueName: \"kubernetes.io/projected/0cda38c1-7ae8-4ccf-a177-21ce6b322104-kube-api-access-x76ld\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.196901 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.196762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cda38c1-7ae8-4ccf-a177-21ce6b322104-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.197008 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.196886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cda38c1-7ae8-4ccf-a177-21ce6b322104-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.297708 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.297621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76ld\" (UniqueName: \"kubernetes.io/projected/0cda38c1-7ae8-4ccf-a177-21ce6b322104-kube-api-access-x76ld\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.297708 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.297684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cda38c1-7ae8-4ccf-a177-21ce6b322104-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.297917 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.297790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cda38c1-7ae8-4ccf-a177-21ce6b322104-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.298630 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.298592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cda38c1-7ae8-4ccf-a177-21ce6b322104-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.300772 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.300745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cda38c1-7ae8-4ccf-a177-21ce6b322104-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.310721 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.310693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76ld\" (UniqueName: \"kubernetes.io/projected/0cda38c1-7ae8-4ccf-a177-21ce6b322104-kube-api-access-x76ld\") pod \"kuadrant-console-plugin-66b5774985-6gj97\" (UID: \"0cda38c1-7ae8-4ccf-a177-21ce6b322104\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.412404 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.412372    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"
Jul 23 18:08:44.612839 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:44.612817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97"]
Jul 23 18:08:44.615163 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:08:44.615137    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cda38c1_7ae8_4ccf_a177_21ce6b322104.slice/crio-cc936538c1199721d7aba91bde8b4bffb45274540a6b71d0a78821e2873f00ba WatchSource:0}: Error finding container cc936538c1199721d7aba91bde8b4bffb45274540a6b71d0a78821e2873f00ba: Status 404 returned error can't find the container with id cc936538c1199721d7aba91bde8b4bffb45274540a6b71d0a78821e2873f00ba
Jul 23 18:08:45.024139 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:45.024082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97" event={"ID":"0cda38c1-7ae8-4ccf-a177-21ce6b322104","Type":"ContainerStarted","Data":"cc936538c1199721d7aba91bde8b4bffb45274540a6b71d0a78821e2873f00ba"}
Jul 23 18:08:46.035174 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:46.035137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt" event={"ID":"8b1c240f-ee89-49cb-b380-7b921dc62bda","Type":"ContainerStarted","Data":"5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac"}
Jul 23 18:08:46.035604 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:46.035286    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:46.068204 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:46.068046    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt" podStartSLOduration=1.8559785309999999 podStartE2EDuration="4.06802391s" totalImagesPullingTime="2.212045379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:46.067034681 +0000 UTC m=+564.857191806" watchObservedRunningTime="2026-07-23 18:08:46.06802391 +0000 UTC m=+564.858181025"
Jul 23 18:08:50.990784 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:50.990753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77t9"
Jul 23 18:08:53.009768 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:53.009737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:55.002769 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.002730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"]
Jul 23 18:08:55.003244 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.002999    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87" podUID="513e3773-e800-4a26-bc48-62e234de0f94" containerName="manager" containerID="cri-o://da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba" gracePeriod=2
Jul 23 18:08:55.021787 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.021754    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"]
Jul 23 18:08:55.024198 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:55.024166    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"513e3773-e800-4a26-bc48-62e234de0f94\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-lpd87\": pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:55.028609 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.028573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kh4pr"
Jul 23 18:08:55.031121 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:55.031085    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="513e3773-e800-4a26-bc48-62e234de0f94" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:55.035880 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.035843    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"]
Jul 23 18:08:55.036146 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.036118    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" containerName="manager" containerID="cri-o://5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac" gracePeriod=2
Jul 23 18:08:55.038192 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.038142    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:55.054578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.052958    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"]
Jul 23 18:08:55.054578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.053400    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e3773-e800-4a26-bc48-62e234de0f94" containerName="manager"
Jul 23 18:08:55.054578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.053415    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e3773-e800-4a26-bc48-62e234de0f94" containerName="manager"
Jul 23 18:08:55.054578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.053515    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="513e3773-e800-4a26-bc48-62e234de0f94" containerName="manager"
Jul 23 18:08:55.057332 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.057306    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"]
Jul 23 18:08:55.057478 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.057420    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.066073 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:55.066033    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:55.072117 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.072088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"]
Jul 23 18:08:55.086860 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.086827    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"]
Jul 23 18:08:55.087378 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.087356    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" containerName="manager"
Jul 23 18:08:55.087378 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.087381    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" containerName="manager"
Jul 23 18:08:55.087536 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.087459    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" containerName="manager"
Jul 23 18:08:55.089323 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.089272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cec81b7-9075-4708-813f-66814daca104-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6dv2\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.089433 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.089350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwdn\" (UniqueName: \"kubernetes.io/projected/0cec81b7-9075-4708-813f-66814daca104-kube-api-access-qlwdn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6dv2\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.092009 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.091912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:08:55.097178 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:55.097143    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="513e3773-e800-4a26-bc48-62e234de0f94" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:55.103008 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.102983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"]
Jul 23 18:08:55.126264 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:55.126232    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:08:55.129956 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:08:55.129909    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="513e3773-e800-4a26-bc48-62e234de0f94" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:08:55.190468 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.190430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cec81b7-9075-4708-813f-66814daca104-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6dv2\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.190468 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.190484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlkk\" (UniqueName: \"kubernetes.io/projected/46616c85-549a-4d9e-a03d-c8cf8bd05b86-kube-api-access-kvlkk\") pod \"limitador-operator-controller-manager-859d7b78bd-9mtnf\" (UID: \"46616c85-549a-4d9e-a03d-c8cf8bd05b86\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:08:55.190749 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.190552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwdn\" (UniqueName: \"kubernetes.io/projected/0cec81b7-9075-4708-813f-66814daca104-kube-api-access-qlwdn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6dv2\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.190888 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.190865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cec81b7-9075-4708-813f-66814daca104-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6dv2\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.201036 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.201010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwdn\" (UniqueName: \"kubernetes.io/projected/0cec81b7-9075-4708-813f-66814daca104-kube-api-access-qlwdn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6dv2\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.291275 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.291238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlkk\" (UniqueName: \"kubernetes.io/projected/46616c85-549a-4d9e-a03d-c8cf8bd05b86-kube-api-access-kvlkk\") pod \"limitador-operator-controller-manager-859d7b78bd-9mtnf\" (UID: \"46616c85-549a-4d9e-a03d-c8cf8bd05b86\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:08:55.316451 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.316422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlkk\" (UniqueName: \"kubernetes.io/projected/46616c85-549a-4d9e-a03d-c8cf8bd05b86-kube-api-access-kvlkk\") pod \"limitador-operator-controller-manager-859d7b78bd-9mtnf\" (UID: \"46616c85-549a-4d9e-a03d-c8cf8bd05b86\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:08:55.439889 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.439849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:08:55.446026 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:55.445994    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:08:57.162794 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.162757    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"]
Jul 23 18:08:57.223646 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.223614    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"]
Jul 23 18:08:57.223832 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.223734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.307698 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.307664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8669\" (UniqueName: \"kubernetes.io/projected/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-kube-api-access-j8669\") pod \"kuadrant-operator-controller-manager-86878c769f-64z9m\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.307861 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.307729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-64z9m\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.408768 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.408728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-64z9m\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.408997 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.408791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8669\" (UniqueName: \"kubernetes.io/projected/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-kube-api-access-j8669\") pod \"kuadrant-operator-controller-manager-86878c769f-64z9m\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.409149 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.409126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-64z9m\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.426542 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.426441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8669\" (UniqueName: \"kubernetes.io/projected/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-kube-api-access-j8669\") pod \"kuadrant-operator-controller-manager-86878c769f-64z9m\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:08:57.536681 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:08:57.536645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:09:06.709052 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.709025    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:09:06.711371 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:06.711332    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:09:06.711646 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.711626    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:09:06.714699 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:06.714642    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:09:06.716926 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:06.716900    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="513e3773-e800-4a26-bc48-62e234de0f94" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:09:06.718660 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:09:06.718633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46616c85_549a_4d9e_a03d_c8cf8bd05b86.slice/crio-bfa721b7657e46d1eeb2f5184eff46051b074b37e93af8dd51b757958947d037 WatchSource:0}: Error finding container bfa721b7657e46d1eeb2f5184eff46051b074b37e93af8dd51b757958947d037: Status 404 returned error can't find the container with id bfa721b7657e46d1eeb2f5184eff46051b074b37e93af8dd51b757958947d037
Jul 23 18:09:06.719233 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.719213    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"]
Jul 23 18:09:06.738833 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.738813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"]
Jul 23 18:09:06.742126 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:09:06.742096    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cec81b7_9075_4708_813f_66814daca104.slice/crio-5e93c5a1cff8b78e2a528a1bfba5d631709f4cb8bce045d89fd0c7dd698b5173 WatchSource:0}: Error finding container 5e93c5a1cff8b78e2a528a1bfba5d631709f4cb8bce045d89fd0c7dd698b5173: Status 404 returned error can't find the container with id 5e93c5a1cff8b78e2a528a1bfba5d631709f4cb8bce045d89fd0c7dd698b5173
Jul 23 18:09:06.765152 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.765126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"]
Jul 23 18:09:06.768798 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:09:06.768774    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268e8bd4_e7f4_44a7_8b51_d9ccf7040535.slice/crio-1d60a8abe943af1836247cacda8c5d2accf7fc4e7110128c7a783ff4008d1bf5 WatchSource:0}: Error finding container 1d60a8abe943af1836247cacda8c5d2accf7fc4e7110128c7a783ff4008d1bf5: Status 404 returned error can't find the container with id 1d60a8abe943af1836247cacda8c5d2accf7fc4e7110128c7a783ff4008d1bf5
Jul 23 18:09:06.798001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.797250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqqvw\" (UniqueName: \"kubernetes.io/projected/513e3773-e800-4a26-bc48-62e234de0f94-kube-api-access-bqqvw\") pod \"513e3773-e800-4a26-bc48-62e234de0f94\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") "
Jul 23 18:09:06.798001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.797305    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/513e3773-e800-4a26-bc48-62e234de0f94-extensions-socket-volume\") pod \"513e3773-e800-4a26-bc48-62e234de0f94\" (UID: \"513e3773-e800-4a26-bc48-62e234de0f94\") "
Jul 23 18:09:06.798001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.797382    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp5vs\" (UniqueName: \"kubernetes.io/projected/8b1c240f-ee89-49cb-b380-7b921dc62bda-kube-api-access-kp5vs\") pod \"8b1c240f-ee89-49cb-b380-7b921dc62bda\" (UID: \"8b1c240f-ee89-49cb-b380-7b921dc62bda\") "
Jul 23 18:09:06.798294 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.798250    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513e3773-e800-4a26-bc48-62e234de0f94-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "513e3773-e800-4a26-bc48-62e234de0f94" (UID: "513e3773-e800-4a26-bc48-62e234de0f94"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:09:06.800577 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.800496    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513e3773-e800-4a26-bc48-62e234de0f94-kube-api-access-bqqvw" (OuterVolumeSpecName: "kube-api-access-bqqvw") pod "513e3773-e800-4a26-bc48-62e234de0f94" (UID: "513e3773-e800-4a26-bc48-62e234de0f94"). InnerVolumeSpecName "kube-api-access-bqqvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:09:06.800656 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.800580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1c240f-ee89-49cb-b380-7b921dc62bda-kube-api-access-kp5vs" (OuterVolumeSpecName: "kube-api-access-kp5vs") pod "8b1c240f-ee89-49cb-b380-7b921dc62bda" (UID: "8b1c240f-ee89-49cb-b380-7b921dc62bda"). InnerVolumeSpecName "kube-api-access-kp5vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:09:06.898460 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.898419    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/513e3773-e800-4a26-bc48-62e234de0f94-extensions-socket-volume\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:06.898460 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.898460    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5vs\" (UniqueName: \"kubernetes.io/projected/8b1c240f-ee89-49cb-b380-7b921dc62bda-kube-api-access-kp5vs\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:06.898699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:06.898474    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqqvw\" (UniqueName: \"kubernetes.io/projected/513e3773-e800-4a26-bc48-62e234de0f94-kube-api-access-bqqvw\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:07.165201 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.165108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m" event={"ID":"268e8bd4-e7f4-44a7-8b51-d9ccf7040535","Type":"ContainerStarted","Data":"98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4"}
Jul 23 18:09:07.165201 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.165155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m" event={"ID":"268e8bd4-e7f4-44a7-8b51-d9ccf7040535","Type":"ContainerStarted","Data":"1d60a8abe943af1836247cacda8c5d2accf7fc4e7110128c7a783ff4008d1bf5"}
Jul 23 18:09:07.165421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.165239    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:09:07.166874 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.166852    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf" event={"ID":"46616c85-549a-4d9e-a03d-c8cf8bd05b86","Type":"ContainerStarted","Data":"acd5bc6410325a1b10ac90cd8e2900661206ce8c35001a97290184922f64e96e"}
Jul 23 18:09:07.167038 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.166886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf" event={"ID":"46616c85-549a-4d9e-a03d-c8cf8bd05b86","Type":"ContainerStarted","Data":"bfa721b7657e46d1eeb2f5184eff46051b074b37e93af8dd51b757958947d037"}
Jul 23 18:09:07.167038 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.166908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:09:07.168119 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.168095    2562 generic.go:356] "Generic (PLEG): container finished" podID="513e3773-e800-4a26-bc48-62e234de0f94" containerID="da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba" exitCode=0
Jul 23 18:09:07.168208 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.168132    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:09:07.168208 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.168172    2562 scope.go:117] "RemoveContainer" containerID="da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba"
Jul 23 18:09:07.169485 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.169467    2562 generic.go:356] "Generic (PLEG): container finished" podID="8b1c240f-ee89-49cb-b380-7b921dc62bda" containerID="5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac" exitCode=0
Jul 23 18:09:07.169560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.169500    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:09:07.171267 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.171244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97" event={"ID":"0cda38c1-7ae8-4ccf-a177-21ce6b322104","Type":"ContainerStarted","Data":"9050fe6cb8c48419f771254a93f5be1f5b008a1dff9c25995d1662c9cb20e276"}
Jul 23 18:09:07.173035 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.173016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2" event={"ID":"0cec81b7-9075-4708-813f-66814daca104","Type":"ContainerStarted","Data":"8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965"}
Jul 23 18:09:07.173132 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.173048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2" event={"ID":"0cec81b7-9075-4708-813f-66814daca104","Type":"ContainerStarted","Data":"5e93c5a1cff8b78e2a528a1bfba5d631709f4cb8bce045d89fd0c7dd698b5173"}
Jul 23 18:09:07.173197 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.173128    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:09:07.178229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.178191    2562 scope.go:117] "RemoveContainer" containerID="da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba"
Jul 23 18:09:07.178567 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:07.178547    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba\": container with ID starting with da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba not found: ID does not exist" containerID="da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba"
Jul 23 18:09:07.178648 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.178578    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba"} err="failed to get container status \"da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba\": rpc error: code = NotFound desc = could not find container \"da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba\": container with ID starting with da93dde8061d71958889cbc83133157087a216b236821b2365eaccd3793f90ba not found: ID does not exist"
Jul 23 18:09:07.178648 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.178600    2562 scope.go:117] "RemoveContainer" containerID="5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac"
Jul 23 18:09:07.186608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.186590    2562 scope.go:117] "RemoveContainer" containerID="5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac"
Jul 23 18:09:07.186881 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:07.186858    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac\": container with ID starting with 5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac not found: ID does not exist" containerID="5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac"
Jul 23 18:09:07.186920 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.186892    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac"} err="failed to get container status \"5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac\": rpc error: code = NotFound desc = could not find container \"5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac\": container with ID starting with 5aae17c859f5e2bab70cdb50e716ff25de0968af2f88ac699fa102e560e1cbac not found: ID does not exist"
Jul 23 18:09:07.213228 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.213174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m" podStartSLOduration=10.213157999 podStartE2EDuration="10.213157999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:07.210863455 +0000 UTC m=+586.001020573" watchObservedRunningTime="2026-07-23 18:09:07.213157999 +0000 UTC m=+586.003315095"
Jul 23 18:09:07.213391 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:07.213295    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:09:07.219417 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:07.219390    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="513e3773-e800-4a26-bc48-62e234de0f94" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:09:07.256996 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.256924    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf" podStartSLOduration=12.256908962 podStartE2EDuration="12.256908962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:07.256593446 +0000 UTC m=+586.046750560" watchObservedRunningTime="2026-07-23 18:09:07.256908962 +0000 UTC m=+586.047066076"
Jul 23 18:09:07.261958 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:07.261918    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6lvtt\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6lvtt"
Jul 23 18:09:07.318484 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.318425    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2" podStartSLOduration=12.318409187 podStartE2EDuration="12.318409187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:07.31682883 +0000 UTC m=+586.106986006" watchObservedRunningTime="2026-07-23 18:09:07.318409187 +0000 UTC m=+586.108566280"
Jul 23 18:09:07.319673 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:07.319637    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lpd87\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="513e3773-e800-4a26-bc48-62e234de0f94" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpd87"
Jul 23 18:09:07.357142 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.357083    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6gj97" podStartSLOduration=1.34691129 podStartE2EDuration="23.357065532s" totalImagesPullingTime="22.010154242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:07.354699769 +0000 UTC m=+586.144856906" watchObservedRunningTime="2026-07-23 18:09:07.357065532 +0000 UTC m=+586.147222647"
Jul 23 18:09:07.610218 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.610187    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513e3773-e800-4a26-bc48-62e234de0f94" path="/var/lib/kubelet/pods/513e3773-e800-4a26-bc48-62e234de0f94/volumes"
Jul 23 18:09:07.610544 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:07.610503    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1c240f-ee89-49cb-b380-7b921dc62bda" path="/var/lib/kubelet/pods/8b1c240f-ee89-49cb-b380-7b921dc62bda/volumes"
Jul 23 18:09:18.182056 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.182021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:09:18.182548 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.182201    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mtnf"
Jul 23 18:09:18.182548 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.182255    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:09:18.389571 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.389535    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"]
Jul 23 18:09:18.389742 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.389719    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2" podUID="0cec81b7-9075-4708-813f-66814daca104" containerName="manager" containerID="cri-o://8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965" gracePeriod=10
Jul 23 18:09:18.651484 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.651460    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:09:18.803251 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.803168    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cec81b7-9075-4708-813f-66814daca104-extensions-socket-volume\") pod \"0cec81b7-9075-4708-813f-66814daca104\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") "
Jul 23 18:09:18.803251 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.803202    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlwdn\" (UniqueName: \"kubernetes.io/projected/0cec81b7-9075-4708-813f-66814daca104-kube-api-access-qlwdn\") pod \"0cec81b7-9075-4708-813f-66814daca104\" (UID: \"0cec81b7-9075-4708-813f-66814daca104\") "
Jul 23 18:09:18.803537 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.803512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec81b7-9075-4708-813f-66814daca104-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0cec81b7-9075-4708-813f-66814daca104" (UID: "0cec81b7-9075-4708-813f-66814daca104"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:09:18.805177 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.805156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cec81b7-9075-4708-813f-66814daca104-kube-api-access-qlwdn" (OuterVolumeSpecName: "kube-api-access-qlwdn") pod "0cec81b7-9075-4708-813f-66814daca104" (UID: "0cec81b7-9075-4708-813f-66814daca104"). InnerVolumeSpecName "kube-api-access-qlwdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:09:18.904331 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.904296    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cec81b7-9075-4708-813f-66814daca104-extensions-socket-volume\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:18.904331 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:18.904322    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlwdn\" (UniqueName: \"kubernetes.io/projected/0cec81b7-9075-4708-813f-66814daca104-kube-api-access-qlwdn\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:19.236180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.236088    2562 generic.go:356] "Generic (PLEG): container finished" podID="0cec81b7-9075-4708-813f-66814daca104" containerID="8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965" exitCode=0
Jul 23 18:09:19.236180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.236138    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"
Jul 23 18:09:19.236180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.236148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2" event={"ID":"0cec81b7-9075-4708-813f-66814daca104","Type":"ContainerDied","Data":"8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965"}
Jul 23 18:09:19.236694 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.236192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2" event={"ID":"0cec81b7-9075-4708-813f-66814daca104","Type":"ContainerDied","Data":"5e93c5a1cff8b78e2a528a1bfba5d631709f4cb8bce045d89fd0c7dd698b5173"}
Jul 23 18:09:19.236694 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.236214    2562 scope.go:117] "RemoveContainer" containerID="8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965"
Jul 23 18:09:19.245607 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.245591    2562 scope.go:117] "RemoveContainer" containerID="8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965"
Jul 23 18:09:19.245856 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:19.245840    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965\": container with ID starting with 8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965 not found: ID does not exist" containerID="8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965"
Jul 23 18:09:19.245902 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.245864    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965"} err="failed to get container status \"8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965\": rpc error: code = NotFound desc = could not find container \"8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965\": container with ID starting with 8ea88ad933e62dd5e550bda2cddb020153d2c5fc76cb80c290d389ca90752965 not found: ID does not exist"
Jul 23 18:09:19.270348 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.269272    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"]
Jul 23 18:09:19.275129 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.275099    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6dv2"]
Jul 23 18:09:19.608403 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:19.608366    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cec81b7-9075-4708-813f-66814daca104" path="/var/lib/kubelet/pods/0cec81b7-9075-4708-813f-66814daca104/volumes"
Jul 23 18:09:21.549673 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:21.549645    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:09:21.550143 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:21.549857    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:09:37.047739 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.047701    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:37.048231 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.048065    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cec81b7-9075-4708-813f-66814daca104" containerName="manager"
Jul 23 18:09:37.048231 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.048078    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cec81b7-9075-4708-813f-66814daca104" containerName="manager"
Jul 23 18:09:37.048231 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.048146    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cec81b7-9075-4708-813f-66814daca104" containerName="manager"
Jul 23 18:09:37.081648 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.081608    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:37.081648 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.081638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.084150 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.084119    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 18:09:37.137159 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.137123    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:37.259311 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.259270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5bfabb46-0272-4b51-af7b-85818523d74f-config-file\") pod \"limitador-limitador-64c94fd64f-whzrk\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.259474 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.259395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtb8\" (UniqueName: \"kubernetes.io/projected/5bfabb46-0272-4b51-af7b-85818523d74f-kube-api-access-qdtb8\") pod \"limitador-limitador-64c94fd64f-whzrk\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.360857 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.360764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtb8\" (UniqueName: \"kubernetes.io/projected/5bfabb46-0272-4b51-af7b-85818523d74f-kube-api-access-qdtb8\") pod \"limitador-limitador-64c94fd64f-whzrk\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.360857 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.360820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5bfabb46-0272-4b51-af7b-85818523d74f-config-file\") pod \"limitador-limitador-64c94fd64f-whzrk\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.361561 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.361540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5bfabb46-0272-4b51-af7b-85818523d74f-config-file\") pod \"limitador-limitador-64c94fd64f-whzrk\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.373833 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.373813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtb8\" (UniqueName: \"kubernetes.io/projected/5bfabb46-0272-4b51-af7b-85818523d74f-kube-api-access-qdtb8\") pod \"limitador-limitador-64c94fd64f-whzrk\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.391647 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.391623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:37.528360 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.528324    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:37.529885 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:09:37.529855    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfabb46_0272_4b51_af7b_85818523d74f.slice/crio-6e2abe49f7eb49893f8629572f71b4bd605d65a8812d71b3d147cfd7fb8135b5 WatchSource:0}: Error finding container 6e2abe49f7eb49893f8629572f71b4bd605d65a8812d71b3d147cfd7fb8135b5: Status 404 returned error can't find the container with id 6e2abe49f7eb49893f8629572f71b4bd605d65a8812d71b3d147cfd7fb8135b5
Jul 23 18:09:37.865456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.865417    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-l7hqd"]
Jul 23 18:09:37.870260 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.870230    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:37.872664 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.872641    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pnkkw\""
Jul 23 18:09:37.881414 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.881388    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7hqd"]
Jul 23 18:09:37.965574 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:37.965537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qmh\" (UniqueName: \"kubernetes.io/projected/914e07ed-2489-4cb8-8adb-06db93aaefc2-kube-api-access-s6qmh\") pod \"authorino-58cd955675-l7hqd\" (UID: \"914e07ed-2489-4cb8-8adb-06db93aaefc2\") " pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:38.067022 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:38.066988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qmh\" (UniqueName: \"kubernetes.io/projected/914e07ed-2489-4cb8-8adb-06db93aaefc2-kube-api-access-s6qmh\") pod \"authorino-58cd955675-l7hqd\" (UID: \"914e07ed-2489-4cb8-8adb-06db93aaefc2\") " pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:38.083749 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:38.083724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qmh\" (UniqueName: \"kubernetes.io/projected/914e07ed-2489-4cb8-8adb-06db93aaefc2-kube-api-access-s6qmh\") pod \"authorino-58cd955675-l7hqd\" (UID: \"914e07ed-2489-4cb8-8adb-06db93aaefc2\") " pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:38.180279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:38.180199    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:38.327663 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:38.327636    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7hqd"]
Jul 23 18:09:38.329902 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:09:38.329861    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod914e07ed_2489_4cb8_8adb_06db93aaefc2.slice/crio-88cc3d755fc4a5cdbab753995ae4b0c46751783ac089fb52ce6a4622ef755e87 WatchSource:0}: Error finding container 88cc3d755fc4a5cdbab753995ae4b0c46751783ac089fb52ce6a4622ef755e87: Status 404 returned error can't find the container with id 88cc3d755fc4a5cdbab753995ae4b0c46751783ac089fb52ce6a4622ef755e87
Jul 23 18:09:38.333310 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:38.333282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk" event={"ID":"5bfabb46-0272-4b51-af7b-85818523d74f","Type":"ContainerStarted","Data":"6e2abe49f7eb49893f8629572f71b4bd605d65a8812d71b3d147cfd7fb8135b5"}
Jul 23 18:09:39.342810 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:39.342768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7hqd" event={"ID":"914e07ed-2489-4cb8-8adb-06db93aaefc2","Type":"ContainerStarted","Data":"88cc3d755fc4a5cdbab753995ae4b0c46751783ac089fb52ce6a4622ef755e87"}
Jul 23 18:09:42.361737 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:42.361700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk" event={"ID":"5bfabb46-0272-4b51-af7b-85818523d74f","Type":"ContainerStarted","Data":"c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15"}
Jul 23 18:09:42.362243 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:42.361795    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:42.363253 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:42.363230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7hqd" event={"ID":"914e07ed-2489-4cb8-8adb-06db93aaefc2","Type":"ContainerStarted","Data":"6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1"}
Jul 23 18:09:42.381497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:42.381445    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk" podStartSLOduration=0.765434971 podStartE2EDuration="5.381430276s" totalImagesPullingTime="4.615995305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:42.379882713 +0000 UTC m=+621.170039828" watchObservedRunningTime="2026-07-23 18:09:42.381430276 +0000 UTC m=+621.171587453"
Jul 23 18:09:42.396968 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:42.396848    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-l7hqd" podStartSLOduration=1.573730675 podStartE2EDuration="5.396833722s" totalImagesPullingTime="3.823103047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:09:42.394974739 +0000 UTC m=+621.185131877" watchObservedRunningTime="2026-07-23 18:09:42.396833722 +0000 UTC m=+621.186990837"
Jul 23 18:09:42.996688 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:42.996656    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7hqd"]
Jul 23 18:09:44.373453 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:44.373312    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-l7hqd" podUID="914e07ed-2489-4cb8-8adb-06db93aaefc2" containerName="authorino" containerID="cri-o://6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1" gracePeriod=30
Jul 23 18:09:44.620167 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:44.620146    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:44.729416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:44.729323    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6qmh\" (UniqueName: \"kubernetes.io/projected/914e07ed-2489-4cb8-8adb-06db93aaefc2-kube-api-access-s6qmh\") pod \"914e07ed-2489-4cb8-8adb-06db93aaefc2\" (UID: \"914e07ed-2489-4cb8-8adb-06db93aaefc2\") "
Jul 23 18:09:44.731536 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:44.731510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914e07ed-2489-4cb8-8adb-06db93aaefc2-kube-api-access-s6qmh" (OuterVolumeSpecName: "kube-api-access-s6qmh") pod "914e07ed-2489-4cb8-8adb-06db93aaefc2" (UID: "914e07ed-2489-4cb8-8adb-06db93aaefc2"). InnerVolumeSpecName "kube-api-access-s6qmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:09:44.830817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:44.830784    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6qmh\" (UniqueName: \"kubernetes.io/projected/914e07ed-2489-4cb8-8adb-06db93aaefc2-kube-api-access-s6qmh\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:45.379843 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.379805    2562 generic.go:356] "Generic (PLEG): container finished" podID="914e07ed-2489-4cb8-8adb-06db93aaefc2" containerID="6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1" exitCode=0
Jul 23 18:09:45.380284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.379861    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7hqd"
Jul 23 18:09:45.380284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.379890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7hqd" event={"ID":"914e07ed-2489-4cb8-8adb-06db93aaefc2","Type":"ContainerDied","Data":"6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1"}
Jul 23 18:09:45.380284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.379932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7hqd" event={"ID":"914e07ed-2489-4cb8-8adb-06db93aaefc2","Type":"ContainerDied","Data":"88cc3d755fc4a5cdbab753995ae4b0c46751783ac089fb52ce6a4622ef755e87"}
Jul 23 18:09:45.380284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.379969    2562 scope.go:117] "RemoveContainer" containerID="6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1"
Jul 23 18:09:45.388592 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.388575    2562 scope.go:117] "RemoveContainer" containerID="6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1"
Jul 23 18:09:45.388854 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:45.388837    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1\": container with ID starting with 6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1 not found: ID does not exist" containerID="6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1"
Jul 23 18:09:45.388888 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.388864    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1"} err="failed to get container status \"6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1\": rpc error: code = NotFound desc = could not find container \"6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1\": container with ID starting with 6015536504ab2c4d115968415667ec62c6ea2144ecb38408332872c544b418c1 not found: ID does not exist"
Jul 23 18:09:45.401972 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.401875    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7hqd"]
Jul 23 18:09:45.403756 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.403739    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7hqd"]
Jul 23 18:09:45.607538 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:45.607503    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="914e07ed-2489-4cb8-8adb-06db93aaefc2" path="/var/lib/kubelet/pods/914e07ed-2489-4cb8-8adb-06db93aaefc2/volumes"
Jul 23 18:09:53.369675 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:53.369644    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:54.012961 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.012915    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:54.013149 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.013127    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk" podUID="5bfabb46-0272-4b51-af7b-85818523d74f" containerName="limitador" containerID="cri-o://c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15" gracePeriod=30
Jul 23 18:09:54.565124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.565097    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:54.610683 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.610612    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdtb8\" (UniqueName: \"kubernetes.io/projected/5bfabb46-0272-4b51-af7b-85818523d74f-kube-api-access-qdtb8\") pod \"5bfabb46-0272-4b51-af7b-85818523d74f\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") "
Jul 23 18:09:54.610683 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.610666    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5bfabb46-0272-4b51-af7b-85818523d74f-config-file\") pod \"5bfabb46-0272-4b51-af7b-85818523d74f\" (UID: \"5bfabb46-0272-4b51-af7b-85818523d74f\") "
Jul 23 18:09:54.611034 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.611010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfabb46-0272-4b51-af7b-85818523d74f-config-file" (OuterVolumeSpecName: "config-file") pod "5bfabb46-0272-4b51-af7b-85818523d74f" (UID: "5bfabb46-0272-4b51-af7b-85818523d74f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:09:54.612754 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.612731    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfabb46-0272-4b51-af7b-85818523d74f-kube-api-access-qdtb8" (OuterVolumeSpecName: "kube-api-access-qdtb8") pod "5bfabb46-0272-4b51-af7b-85818523d74f" (UID: "5bfabb46-0272-4b51-af7b-85818523d74f"). InnerVolumeSpecName "kube-api-access-qdtb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:09:54.712091 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.712058    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdtb8\" (UniqueName: \"kubernetes.io/projected/5bfabb46-0272-4b51-af7b-85818523d74f-kube-api-access-qdtb8\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:54.712091 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:54.712084    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5bfabb46-0272-4b51-af7b-85818523d74f-config-file\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:09:55.430803 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.430765    2562 generic.go:356] "Generic (PLEG): container finished" podID="5bfabb46-0272-4b51-af7b-85818523d74f" containerID="c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15" exitCode=0
Jul 23 18:09:55.430803 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.430808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk" event={"ID":"5bfabb46-0272-4b51-af7b-85818523d74f","Type":"ContainerDied","Data":"c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15"}
Jul 23 18:09:55.431053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.430811    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk"
Jul 23 18:09:55.431053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.430859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-whzrk" event={"ID":"5bfabb46-0272-4b51-af7b-85818523d74f","Type":"ContainerDied","Data":"6e2abe49f7eb49893f8629572f71b4bd605d65a8812d71b3d147cfd7fb8135b5"}
Jul 23 18:09:55.431053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.430885    2562 scope.go:117] "RemoveContainer" containerID="c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15"
Jul 23 18:09:55.440313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.440292    2562 scope.go:117] "RemoveContainer" containerID="c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15"
Jul 23 18:09:55.440582 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:09:55.440563    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15\": container with ID starting with c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15 not found: ID does not exist" containerID="c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15"
Jul 23 18:09:55.440638 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.440595    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15"} err="failed to get container status \"c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15\": rpc error: code = NotFound desc = could not find container \"c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15\": container with ID starting with c141ee245e676e921ba5f062a0e99bfea80c705610668fffdcb84e3b1008df15 not found: ID does not exist"
Jul 23 18:09:55.459635 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.459603    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:55.461620 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.461599    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-whzrk"]
Jul 23 18:09:55.607673 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:55.607641    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfabb46-0272-4b51-af7b-85818523d74f" path="/var/lib/kubelet/pods/5bfabb46-0272-4b51-af7b-85818523d74f/volumes"
Jul 23 18:09:58.801262 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801225    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"]
Jul 23 18:09:58.801652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801562    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914e07ed-2489-4cb8-8adb-06db93aaefc2" containerName="authorino"
Jul 23 18:09:58.801652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801573    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="914e07ed-2489-4cb8-8adb-06db93aaefc2" containerName="authorino"
Jul 23 18:09:58.801652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801582    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfabb46-0272-4b51-af7b-85818523d74f" containerName="limitador"
Jul 23 18:09:58.801652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801587    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfabb46-0272-4b51-af7b-85818523d74f" containerName="limitador"
Jul 23 18:09:58.801652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801635    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfabb46-0272-4b51-af7b-85818523d74f" containerName="limitador"
Jul 23 18:09:58.801652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.801643    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="914e07ed-2489-4cb8-8adb-06db93aaefc2" containerName="authorino"
Jul 23 18:09:58.805983 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.805955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:58.808335 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.808311    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 23 18:09:58.808335 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.808331    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ql5tr\""
Jul 23 18:09:58.809138 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.809122    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:09:58.809213 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.809127    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:09:58.827881 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.827857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"]
Jul 23 18:09:58.947933 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.947898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a287be2e-5a18-4abf-80af-01aca49df024-data\") pod \"postgres-69fb4594cf-ccqzj\" (UID: \"a287be2e-5a18-4abf-80af-01aca49df024\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:58.948152 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:58.947970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h5z6\" (UniqueName: \"kubernetes.io/projected/a287be2e-5a18-4abf-80af-01aca49df024-kube-api-access-9h5z6\") pod \"postgres-69fb4594cf-ccqzj\" (UID: \"a287be2e-5a18-4abf-80af-01aca49df024\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:59.049451 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.049411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a287be2e-5a18-4abf-80af-01aca49df024-data\") pod \"postgres-69fb4594cf-ccqzj\" (UID: \"a287be2e-5a18-4abf-80af-01aca49df024\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:59.049619 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.049480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h5z6\" (UniqueName: \"kubernetes.io/projected/a287be2e-5a18-4abf-80af-01aca49df024-kube-api-access-9h5z6\") pod \"postgres-69fb4594cf-ccqzj\" (UID: \"a287be2e-5a18-4abf-80af-01aca49df024\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:59.049888 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.049871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a287be2e-5a18-4abf-80af-01aca49df024-data\") pod \"postgres-69fb4594cf-ccqzj\" (UID: \"a287be2e-5a18-4abf-80af-01aca49df024\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:59.060578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.060508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h5z6\" (UniqueName: \"kubernetes.io/projected/a287be2e-5a18-4abf-80af-01aca49df024-kube-api-access-9h5z6\") pod \"postgres-69fb4594cf-ccqzj\" (UID: \"a287be2e-5a18-4abf-80af-01aca49df024\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:59.117639 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.117607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:09:59.266327 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.266287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"]
Jul 23 18:09:59.268594 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:09:59.268565    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda287be2e_5a18_4abf_80af_01aca49df024.slice/crio-1060d3e4393a79c794051d4904d56760530d5c5eb57076fadbed0de7c5dbf10e WatchSource:0}: Error finding container 1060d3e4393a79c794051d4904d56760530d5c5eb57076fadbed0de7c5dbf10e: Status 404 returned error can't find the container with id 1060d3e4393a79c794051d4904d56760530d5c5eb57076fadbed0de7c5dbf10e
Jul 23 18:09:59.458719 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:09:59.458632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj" event={"ID":"a287be2e-5a18-4abf-80af-01aca49df024","Type":"ContainerStarted","Data":"1060d3e4393a79c794051d4904d56760530d5c5eb57076fadbed0de7c5dbf10e"}
Jul 23 18:10:04.489743 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:04.489660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj" event={"ID":"a287be2e-5a18-4abf-80af-01aca49df024","Type":"ContainerStarted","Data":"d915f1965192274b80c67880a59df6172c6ef3e2f3d31bf0792dd156cf04bc8d"}
Jul 23 18:10:04.490114 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:04.489747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:10:04.518547 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:04.518490    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj" podStartSLOduration=1.727806438 podStartE2EDuration="6.518466396s" totalImagesPullingTime="4.790659958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:04.517179972 +0000 UTC m=+643.307337087" watchObservedRunningTime="2026-07-23 18:10:04.518466396 +0000 UTC m=+643.308623510"
Jul 23 18:10:10.525559 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:10.525527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ccqzj"
Jul 23 18:10:11.084704 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.084669    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jkmxf"]
Jul 23 18:10:11.088689 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.088668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:11.091125 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.091104    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pnkkw\""
Jul 23 18:10:11.099539 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.099511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jkmxf"]
Jul 23 18:10:11.256701 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.256662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48564\" (UniqueName: \"kubernetes.io/projected/6da820d3-68a1-46bf-b453-ad4321edabe4-kube-api-access-48564\") pod \"authorino-5d4b67c446-jkmxf\" (UID: \"6da820d3-68a1-46bf-b453-ad4321edabe4\") " pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:11.350145 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.350056    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jkmxf"]
Jul 23 18:10:11.350321 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:10:11.350302    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-48564], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-jkmxf" podUID="6da820d3-68a1-46bf-b453-ad4321edabe4"
Jul 23 18:10:11.358036 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.358012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48564\" (UniqueName: \"kubernetes.io/projected/6da820d3-68a1-46bf-b453-ad4321edabe4-kube-api-access-48564\") pod \"authorino-5d4b67c446-jkmxf\" (UID: \"6da820d3-68a1-46bf-b453-ad4321edabe4\") " pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:11.376077 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.376052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48564\" (UniqueName: \"kubernetes.io/projected/6da820d3-68a1-46bf-b453-ad4321edabe4-kube-api-access-48564\") pod \"authorino-5d4b67c446-jkmxf\" (UID: \"6da820d3-68a1-46bf-b453-ad4321edabe4\") " pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:11.384119 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.384094    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ffb4688c7-49qt8"]
Jul 23 18:10:11.388010 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.387993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:11.397602 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.397579    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ffb4688c7-49qt8"]
Jul 23 18:10:11.423356 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.423330    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ffb4688c7-49qt8"]
Jul 23 18:10:11.423579 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:10:11.423551    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t4qg9], unattached volumes=[], failed to process volumes=[kube-api-access-t4qg9]: context canceled" pod="kuadrant-system/authorino-5ffb4688c7-49qt8" podUID="4d7e9994-1707-4fed-b023-2a1f7a321681"
Jul 23 18:10:11.460225 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.460199    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d5c65d7b9-9vpdh"]
Jul 23 18:10:11.463973 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.463927    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.466253 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.466231    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 23 18:10:11.472455 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.472435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d5c65d7b9-9vpdh"]
Jul 23 18:10:11.526422 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.526389    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:11.526831 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.526389    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:11.533747 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.533723    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:11.536640 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.536616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:11.559209 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.559178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qg9\" (UniqueName: \"kubernetes.io/projected/4d7e9994-1707-4fed-b023-2a1f7a321681-kube-api-access-t4qg9\") pod \"authorino-5ffb4688c7-49qt8\" (UID: \"4d7e9994-1707-4fed-b023-2a1f7a321681\") " pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:11.559355 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.559222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa5f8707-671a-4fe8-b1cd-25e11cea4073-tls-cert\") pod \"authorino-7d5c65d7b9-9vpdh\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") " pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.559355 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.559240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vk6d\" (UniqueName: \"kubernetes.io/projected/fa5f8707-671a-4fe8-b1cd-25e11cea4073-kube-api-access-4vk6d\") pod \"authorino-7d5c65d7b9-9vpdh\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") " pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.660388 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.660356    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48564\" (UniqueName: \"kubernetes.io/projected/6da820d3-68a1-46bf-b453-ad4321edabe4-kube-api-access-48564\") pod \"6da820d3-68a1-46bf-b453-ad4321edabe4\" (UID: \"6da820d3-68a1-46bf-b453-ad4321edabe4\") "
Jul 23 18:10:11.660546 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.660501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qg9\" (UniqueName: \"kubernetes.io/projected/4d7e9994-1707-4fed-b023-2a1f7a321681-kube-api-access-t4qg9\") pod \"authorino-5ffb4688c7-49qt8\" (UID: \"4d7e9994-1707-4fed-b023-2a1f7a321681\") " pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:11.660546 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.660537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa5f8707-671a-4fe8-b1cd-25e11cea4073-tls-cert\") pod \"authorino-7d5c65d7b9-9vpdh\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") " pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.660661 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.660554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vk6d\" (UniqueName: \"kubernetes.io/projected/fa5f8707-671a-4fe8-b1cd-25e11cea4073-kube-api-access-4vk6d\") pod \"authorino-7d5c65d7b9-9vpdh\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") " pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.662715 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.662682    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da820d3-68a1-46bf-b453-ad4321edabe4-kube-api-access-48564" (OuterVolumeSpecName: "kube-api-access-48564") pod "6da820d3-68a1-46bf-b453-ad4321edabe4" (UID: "6da820d3-68a1-46bf-b453-ad4321edabe4"). InnerVolumeSpecName "kube-api-access-48564". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:10:11.662897 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.662879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa5f8707-671a-4fe8-b1cd-25e11cea4073-tls-cert\") pod \"authorino-7d5c65d7b9-9vpdh\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") " pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.669247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.669227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qg9\" (UniqueName: \"kubernetes.io/projected/4d7e9994-1707-4fed-b023-2a1f7a321681-kube-api-access-t4qg9\") pod \"authorino-5ffb4688c7-49qt8\" (UID: \"4d7e9994-1707-4fed-b023-2a1f7a321681\") " pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:11.669336 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.669282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vk6d\" (UniqueName: \"kubernetes.io/projected/fa5f8707-671a-4fe8-b1cd-25e11cea4073-kube-api-access-4vk6d\") pod \"authorino-7d5c65d7b9-9vpdh\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") " pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.761147 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.761113    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4qg9\" (UniqueName: \"kubernetes.io/projected/4d7e9994-1707-4fed-b023-2a1f7a321681-kube-api-access-t4qg9\") pod \"4d7e9994-1707-4fed-b023-2a1f7a321681\" (UID: \"4d7e9994-1707-4fed-b023-2a1f7a321681\") "
Jul 23 18:10:11.761304 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.761249    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48564\" (UniqueName: \"kubernetes.io/projected/6da820d3-68a1-46bf-b453-ad4321edabe4-kube-api-access-48564\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:10:11.763279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.763248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7e9994-1707-4fed-b023-2a1f7a321681-kube-api-access-t4qg9" (OuterVolumeSpecName: "kube-api-access-t4qg9") pod "4d7e9994-1707-4fed-b023-2a1f7a321681" (UID: "4d7e9994-1707-4fed-b023-2a1f7a321681"). InnerVolumeSpecName "kube-api-access-t4qg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:10:11.773415 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.773375    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:10:11.863976 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.863154    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qg9\" (UniqueName: \"kubernetes.io/projected/4d7e9994-1707-4fed-b023-2a1f7a321681-kube-api-access-t4qg9\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:10:11.944330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:11.944249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d5c65d7b9-9vpdh"]
Jul 23 18:10:11.951618 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:10:11.951589    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa5f8707_671a_4fe8_b1cd_25e11cea4073.slice/crio-e67baa06cc350022b46dfd5095fa9ec500a5a05c6a2fac3578c4daccbbb0a6ca WatchSource:0}: Error finding container e67baa06cc350022b46dfd5095fa9ec500a5a05c6a2fac3578c4daccbbb0a6ca: Status 404 returned error can't find the container with id e67baa06cc350022b46dfd5095fa9ec500a5a05c6a2fac3578c4daccbbb0a6ca
Jul 23 18:10:12.533149 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.533112    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh" event={"ID":"fa5f8707-671a-4fe8-b1cd-25e11cea4073","Type":"ContainerStarted","Data":"da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f"}
Jul 23 18:10:12.533527 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.533155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh" event={"ID":"fa5f8707-671a-4fe8-b1cd-25e11cea4073","Type":"ContainerStarted","Data":"e67baa06cc350022b46dfd5095fa9ec500a5a05c6a2fac3578c4daccbbb0a6ca"}
Jul 23 18:10:12.533527 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.533163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jkmxf"
Jul 23 18:10:12.533527 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.533338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb4688c7-49qt8"
Jul 23 18:10:12.550907 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.550859    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh" podStartSLOduration=1.1783939669999999 podStartE2EDuration="1.550843601s" totalImagesPullingTime="372.449634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:12.548735494 +0000 UTC m=+651.338892608" watchObservedRunningTime="2026-07-23 18:10:12.550843601 +0000 UTC m=+651.341000716"
Jul 23 18:10:12.585444 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.585355    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jkmxf"]
Jul 23 18:10:12.595870 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.595840    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jkmxf"]
Jul 23 18:10:12.619204 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.619177    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ffb4688c7-49qt8"]
Jul 23 18:10:12.623771 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:12.623745    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ffb4688c7-49qt8"]
Jul 23 18:10:13.607724 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:13.607691    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7e9994-1707-4fed-b023-2a1f7a321681" path="/var/lib/kubelet/pods/4d7e9994-1707-4fed-b023-2a1f7a321681/volumes"
Jul 23 18:10:13.608105 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:13.607912    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6da820d3-68a1-46bf-b453-ad4321edabe4" path="/var/lib/kubelet/pods/6da820d3-68a1-46bf-b453-ad4321edabe4/volumes"
Jul 23 18:10:17.730416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.730375    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-jnjtl"]
Jul 23 18:10:17.825064 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.825029    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-jnjtl"]
Jul 23 18:10:17.825269 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.825139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:17.828010 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.827988    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 18:10:17.828893 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.828865    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 18:10:17.829029 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.829015    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p7688\""
Jul 23 18:10:17.912643 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.912612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxrq\" (UniqueName: \"kubernetes.io/projected/cd1f044f-a652-4d35-8df7-07f5bf87f45b-kube-api-access-tfxrq\") pod \"maas-controller-86bcf48f9b-jnjtl\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") " pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:17.912812 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:17.912667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1f044f-a652-4d35-8df7-07f5bf87f45b-webhook-cert\") pod \"maas-controller-86bcf48f9b-jnjtl\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") " pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:18.013977 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.013928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1f044f-a652-4d35-8df7-07f5bf87f45b-webhook-cert\") pod \"maas-controller-86bcf48f9b-jnjtl\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") " pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:18.014124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.014035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxrq\" (UniqueName: \"kubernetes.io/projected/cd1f044f-a652-4d35-8df7-07f5bf87f45b-kube-api-access-tfxrq\") pod \"maas-controller-86bcf48f9b-jnjtl\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") " pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:18.016501 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.016479    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1f044f-a652-4d35-8df7-07f5bf87f45b-webhook-cert\") pod \"maas-controller-86bcf48f9b-jnjtl\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") " pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:18.024291 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.024269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxrq\" (UniqueName: \"kubernetes.io/projected/cd1f044f-a652-4d35-8df7-07f5bf87f45b-kube-api-access-tfxrq\") pod \"maas-controller-86bcf48f9b-jnjtl\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") " pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:18.135084 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.135046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:18.267915 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.267891    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-jnjtl"]
Jul 23 18:10:18.269644 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:10:18.269618    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1f044f_a652_4d35_8df7_07f5bf87f45b.slice/crio-a61b1031e1232355de2003a88f78c52a31fc7c748a43aae94dd905b6947592b4 WatchSource:0}: Error finding container a61b1031e1232355de2003a88f78c52a31fc7c748a43aae94dd905b6947592b4: Status 404 returned error can't find the container with id a61b1031e1232355de2003a88f78c52a31fc7c748a43aae94dd905b6947592b4
Jul 23 18:10:18.565755 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:18.565668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl" event={"ID":"cd1f044f-a652-4d35-8df7-07f5bf87f45b","Type":"ContainerStarted","Data":"a61b1031e1232355de2003a88f78c52a31fc7c748a43aae94dd905b6947592b4"}
Jul 23 18:10:20.584336 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:20.584302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl" event={"ID":"cd1f044f-a652-4d35-8df7-07f5bf87f45b","Type":"ContainerStarted","Data":"0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3"}
Jul 23 18:10:20.584814 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:20.584433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:20.619191 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:20.619136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl" podStartSLOduration=1.447552422 podStartE2EDuration="3.619118838s" totalImagesPullingTime="2.171566416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:20.617653271 +0000 UTC m=+659.407810388" watchObservedRunningTime="2026-07-23 18:10:20.619118838 +0000 UTC m=+659.409275953"
Jul 23 18:10:31.591551 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:31.591516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:10:52.291895 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.291860    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"]
Jul 23 18:10:52.295471 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.295446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.298059 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.298026    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 23 18:10:52.298059 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.298042    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rm8hz\""
Jul 23 18:10:52.306877 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.306849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"]
Jul 23 18:10:52.374267 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.374209    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-wcvnv"]
Jul 23 18:10:52.377595 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.377566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.390701 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.390680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lwk\" (UniqueName: \"kubernetes.io/projected/f8c4aab0-e19b-4717-98bf-0d0bccfc76e6-kube-api-access-59lwk\") pod \"payload-pre-processing-7677b774bb-rl2d4\" (UID: \"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.390826 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.390738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c4aab0-e19b-4717-98bf-0d0bccfc76e6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-rl2d4\" (UID: \"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.395837 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.395809    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-wcvnv"]
Jul 23 18:10:52.492048 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.492016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdw5\" (UniqueName: \"kubernetes.io/projected/fa981909-b02a-4eb8-aff2-3219f398b491-kube-api-access-dhdw5\") pod \"payload-processing-5bf47cd988-wcvnv\" (UID: \"fa981909-b02a-4eb8-aff2-3219f398b491\") " pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.492219 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.492074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lwk\" (UniqueName: \"kubernetes.io/projected/f8c4aab0-e19b-4717-98bf-0d0bccfc76e6-kube-api-access-59lwk\") pod \"payload-pre-processing-7677b774bb-rl2d4\" (UID: \"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.492219 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.492178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa981909-b02a-4eb8-aff2-3219f398b491-plugins-config-volume\") pod \"payload-processing-5bf47cd988-wcvnv\" (UID: \"fa981909-b02a-4eb8-aff2-3219f398b491\") " pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.492296 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.492224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c4aab0-e19b-4717-98bf-0d0bccfc76e6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-rl2d4\" (UID: \"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.492843 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.492824    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c4aab0-e19b-4717-98bf-0d0bccfc76e6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-rl2d4\" (UID: \"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.506060 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.506036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lwk\" (UniqueName: \"kubernetes.io/projected/f8c4aab0-e19b-4717-98bf-0d0bccfc76e6-kube-api-access-59lwk\") pod \"payload-pre-processing-7677b774bb-rl2d4\" (UID: \"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.593522 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.593450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa981909-b02a-4eb8-aff2-3219f398b491-plugins-config-volume\") pod \"payload-processing-5bf47cd988-wcvnv\" (UID: \"fa981909-b02a-4eb8-aff2-3219f398b491\") " pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.593522 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.593499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdw5\" (UniqueName: \"kubernetes.io/projected/fa981909-b02a-4eb8-aff2-3219f398b491-kube-api-access-dhdw5\") pod \"payload-processing-5bf47cd988-wcvnv\" (UID: \"fa981909-b02a-4eb8-aff2-3219f398b491\") " pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.594254 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.594228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa981909-b02a-4eb8-aff2-3219f398b491-plugins-config-volume\") pod \"payload-processing-5bf47cd988-wcvnv\" (UID: \"fa981909-b02a-4eb8-aff2-3219f398b491\") " pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.605252 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.605222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdw5\" (UniqueName: \"kubernetes.io/projected/fa981909-b02a-4eb8-aff2-3219f398b491-kube-api-access-dhdw5\") pod \"payload-processing-5bf47cd988-wcvnv\" (UID: \"fa981909-b02a-4eb8-aff2-3219f398b491\") " pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.607074 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.607047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:52.688097 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.688069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:52.772411 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.772385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"]
Jul 23 18:10:52.773818 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:10:52.773789    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c4aab0_e19b_4717_98bf_0d0bccfc76e6.slice/crio-11503c269bc7ef2fcbcd05a252282118c1c9b2bc2be63f974831a39b0294dddd WatchSource:0}: Error finding container 11503c269bc7ef2fcbcd05a252282118c1c9b2bc2be63f974831a39b0294dddd: Status 404 returned error can't find the container with id 11503c269bc7ef2fcbcd05a252282118c1c9b2bc2be63f974831a39b0294dddd
Jul 23 18:10:52.775774 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.775758    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:10:52.859737 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:52.859535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-wcvnv"]
Jul 23 18:10:52.862478 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:10:52.862446    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa981909_b02a_4eb8_aff2_3219f398b491.slice/crio-9370f8389d8020c7dd1b45b512c8dc5c1139628259e7480f3c0f0a5537a9f452 WatchSource:0}: Error finding container 9370f8389d8020c7dd1b45b512c8dc5c1139628259e7480f3c0f0a5537a9f452: Status 404 returned error can't find the container with id 9370f8389d8020c7dd1b45b512c8dc5c1139628259e7480f3c0f0a5537a9f452
Jul 23 18:10:53.762415 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:53.762370    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv" event={"ID":"fa981909-b02a-4eb8-aff2-3219f398b491","Type":"ContainerStarted","Data":"9370f8389d8020c7dd1b45b512c8dc5c1139628259e7480f3c0f0a5537a9f452"}
Jul 23 18:10:53.765250 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:53.765217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4" event={"ID":"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6","Type":"ContainerStarted","Data":"11503c269bc7ef2fcbcd05a252282118c1c9b2bc2be63f974831a39b0294dddd"}
Jul 23 18:10:54.772508 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:54.772474    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv" event={"ID":"fa981909-b02a-4eb8-aff2-3219f398b491","Type":"ContainerStarted","Data":"9d88ad380f316acd561685566197e9fab4d105d57efcfa8a07afae6c1c34b869"}
Jul 23 18:10:54.773011 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:54.772642    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:10:54.774015 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:54.773994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4" event={"ID":"f8c4aab0-e19b-4717-98bf-0d0bccfc76e6","Type":"ContainerStarted","Data":"183d9ee2f8ea8313979a8f89f62ec177307f4ee6abc454c58c314411f3262804"}
Jul 23 18:10:54.774098 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:54.774040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:10:54.792761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:54.792715    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv" podStartSLOduration=1.187801244 podStartE2EDuration="2.7927018s" totalImagesPullingTime="1.604900556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:54.790224411 +0000 UTC m=+693.580381526" watchObservedRunningTime="2026-07-23 18:10:54.7927018 +0000 UTC m=+693.582858926"
Jul 23 18:10:54.809169 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:10:54.809115    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4" podStartSLOduration=1.122153497 podStartE2EDuration="2.809095298s" totalImagesPullingTime="1.686941801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:54.807433561 +0000 UTC m=+693.597590698" watchObservedRunningTime="2026-07-23 18:10:54.809095298 +0000 UTC m=+693.599252416"
Jul 23 18:11:05.780868 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:05.780838    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-wcvnv"
Jul 23 18:11:05.781332 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:05.781084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-rl2d4"
Jul 23 18:11:28.261583 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.261504    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"]
Jul 23 18:11:28.268055 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.268034    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.270567 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.270548    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qc4rc\""
Jul 23 18:11:28.270836 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.270821    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 18:11:28.271413 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.271395    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 18:11:28.271572 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.271553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 23 18:11:28.280216 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.280194    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"]
Jul 23 18:11:28.400606 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.400559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.400774 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.400641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.400774 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.400669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9595162f-0a58-4998-bd90-8aaa1a2856aa-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.400857 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.400762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5nw\" (UniqueName: \"kubernetes.io/projected/9595162f-0a58-4998-bd90-8aaa1a2856aa-kube-api-access-8b5nw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.400857 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.400824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.400921 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.400867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.501684 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.501642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.501869 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.501696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.501869 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.501756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.501869 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.501802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.501869 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.501827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9595162f-0a58-4998-bd90-8aaa1a2856aa-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.502102 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.501884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5nw\" (UniqueName: \"kubernetes.io/projected/9595162f-0a58-4998-bd90-8aaa1a2856aa-kube-api-access-8b5nw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.502157 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.502102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.502210 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.502170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.502246 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.502200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.504256 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.504226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9595162f-0a58-4998-bd90-8aaa1a2856aa-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.504460 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.504441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9595162f-0a58-4998-bd90-8aaa1a2856aa-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.510777 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.510753    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5nw\" (UniqueName: \"kubernetes.io/projected/9595162f-0a58-4998-bd90-8aaa1a2856aa-kube-api-access-8b5nw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bpn5w\" (UID: \"9595162f-0a58-4998-bd90-8aaa1a2856aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.579398 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.579306    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:28.726525 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.726485    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"]
Jul 23 18:11:28.726644 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:11:28.726589    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9595162f_0a58_4998_bd90_8aaa1a2856aa.slice/crio-ff7bdff7544a4d8b3e2ec1f4e3ff63cb01e4f760bcfc7696d3e167a221e5f32a WatchSource:0}: Error finding container ff7bdff7544a4d8b3e2ec1f4e3ff63cb01e4f760bcfc7696d3e167a221e5f32a: Status 404 returned error can't find the container with id ff7bdff7544a4d8b3e2ec1f4e3ff63cb01e4f760bcfc7696d3e167a221e5f32a
Jul 23 18:11:28.957001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:28.956909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w" event={"ID":"9595162f-0a58-4998-bd90-8aaa1a2856aa","Type":"ContainerStarted","Data":"ff7bdff7544a4d8b3e2ec1f4e3ff63cb01e4f760bcfc7696d3e167a221e5f32a"}
Jul 23 18:11:30.970141 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:30.970108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w" event={"ID":"9595162f-0a58-4998-bd90-8aaa1a2856aa","Type":"ContainerStarted","Data":"f19450846c4127f1309214bde724c6925a3fafb513512051736d0f45c25fcf09"}
Jul 23 18:11:30.991688 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:30.991642    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w" podStartSLOduration=1.725911009 podStartE2EDuration="2.991626668s" totalImagesPullingTime="1.265715659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:30.990145426 +0000 UTC m=+729.780302540" watchObservedRunningTime="2026-07-23 18:11:30.991626668 +0000 UTC m=+729.781783782"
Jul 23 18:11:38.579484 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.579436    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:38.579978 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.579516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:38.596568 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.596541    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:38.773832 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.773797    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"]
Jul 23 18:11:38.777896 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.777871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.780487 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.780466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 23 18:11:38.782631 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.782608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.782755 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.782648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflrk\" (UniqueName: \"kubernetes.io/projected/b30fabef-45f0-4488-a849-8e0901dab260-kube-api-access-kflrk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.782755 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.782705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.782848 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.782805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30fabef-45f0-4488-a849-8e0901dab260-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.782848 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.782829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.782920 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.782867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.793513 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.793463    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"]
Jul 23 18:11:38.883505 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.883407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.883505 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.883444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflrk\" (UniqueName: \"kubernetes.io/projected/b30fabef-45f0-4488-a849-8e0901dab260-kube-api-access-kflrk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.883505 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.883470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.883819 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.883525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30fabef-45f0-4488-a849-8e0901dab260-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.883819 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.883542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.883819 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.883578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.884031 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.884003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.884321 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.884044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.884321 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.884068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.885927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.885898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30fabef-45f0-4488-a849-8e0901dab260-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.886206 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.886185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30fabef-45f0-4488-a849-8e0901dab260-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:38.892372 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:38.892349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflrk\" (UniqueName: \"kubernetes.io/projected/b30fabef-45f0-4488-a849-8e0901dab260-kube-api-access-kflrk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rssn4\" (UID: \"b30fabef-45f0-4488-a849-8e0901dab260\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:39.021994 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:39.021967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bpn5w"
Jul 23 18:11:39.088816 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:39.088775    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:39.245307 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:39.245281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"]
Jul 23 18:11:39.247181 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:11:39.247142    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30fabef_45f0_4488_a849_8e0901dab260.slice/crio-ef4764d8e07d7115987bfabc5f3cfae164be7dd2e9f0aa87f9953132b7aafe29 WatchSource:0}: Error finding container ef4764d8e07d7115987bfabc5f3cfae164be7dd2e9f0aa87f9953132b7aafe29: Status 404 returned error can't find the container with id ef4764d8e07d7115987bfabc5f3cfae164be7dd2e9f0aa87f9953132b7aafe29
Jul 23 18:11:40.018638 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:40.018599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4" event={"ID":"b30fabef-45f0-4488-a849-8e0901dab260","Type":"ContainerStarted","Data":"ddfe92bcb73fbe19028321d997f3c6e79839613d8bff50eff86ba0888748b4e7"}
Jul 23 18:11:40.019139 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:40.018663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4" event={"ID":"b30fabef-45f0-4488-a849-8e0901dab260","Type":"ContainerStarted","Data":"ef4764d8e07d7115987bfabc5f3cfae164be7dd2e9f0aa87f9953132b7aafe29"}
Jul 23 18:11:40.040838 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:40.040785    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4" podStartSLOduration=1.777756831 podStartE2EDuration="2.040768338s" totalImagesPullingTime="263.011507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:40.038225033 +0000 UTC m=+738.828382150" watchObservedRunningTime="2026-07-23 18:11:40.040768338 +0000 UTC m=+738.830925483"
Jul 23 18:11:49.089442 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:49.089410    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:49.090063 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:49.089474    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:49.101687 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:49.101663    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:11:50.080279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:11:50.080251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rssn4"
Jul 23 18:13:06.925894 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:06.925848    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d5c65d7b9-9vpdh"]
Jul 23 18:13:06.926466 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:06.926066    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh" podUID="fa5f8707-671a-4fe8-b1cd-25e11cea4073" containerName="authorino" containerID="cri-o://da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f" gracePeriod=30
Jul 23 18:13:07.181448 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.181390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:13:07.274273 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.274239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vk6d\" (UniqueName: \"kubernetes.io/projected/fa5f8707-671a-4fe8-b1cd-25e11cea4073-kube-api-access-4vk6d\") pod \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") "
Jul 23 18:13:07.274441 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.274357    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa5f8707-671a-4fe8-b1cd-25e11cea4073-tls-cert\") pod \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\" (UID: \"fa5f8707-671a-4fe8-b1cd-25e11cea4073\") "
Jul 23 18:13:07.276537 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.276507    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5f8707-671a-4fe8-b1cd-25e11cea4073-kube-api-access-4vk6d" (OuterVolumeSpecName: "kube-api-access-4vk6d") pod "fa5f8707-671a-4fe8-b1cd-25e11cea4073" (UID: "fa5f8707-671a-4fe8-b1cd-25e11cea4073"). InnerVolumeSpecName "kube-api-access-4vk6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:13:07.284764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.284739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5f8707-671a-4fe8-b1cd-25e11cea4073-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fa5f8707-671a-4fe8-b1cd-25e11cea4073" (UID: "fa5f8707-671a-4fe8-b1cd-25e11cea4073"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:13:07.375903 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.375868    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vk6d\" (UniqueName: \"kubernetes.io/projected/fa5f8707-671a-4fe8-b1cd-25e11cea4073-kube-api-access-4vk6d\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:13:07.375903 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.375895    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa5f8707-671a-4fe8-b1cd-25e11cea4073-tls-cert\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:13:07.481467 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.481390    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa5f8707-671a-4fe8-b1cd-25e11cea4073" containerID="da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f" exitCode=0
Jul 23 18:13:07.481467 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.481438    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh"
Jul 23 18:13:07.481639 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.481471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh" event={"ID":"fa5f8707-671a-4fe8-b1cd-25e11cea4073","Type":"ContainerDied","Data":"da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f"}
Jul 23 18:13:07.481639 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.481534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5c65d7b9-9vpdh" event={"ID":"fa5f8707-671a-4fe8-b1cd-25e11cea4073","Type":"ContainerDied","Data":"e67baa06cc350022b46dfd5095fa9ec500a5a05c6a2fac3578c4daccbbb0a6ca"}
Jul 23 18:13:07.481639 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.481555    2562 scope.go:117] "RemoveContainer" containerID="da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f"
Jul 23 18:13:07.493617 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.493596    2562 scope.go:117] "RemoveContainer" containerID="da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f"
Jul 23 18:13:07.493868 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:13:07.493850    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f\": container with ID starting with da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f not found: ID does not exist" containerID="da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f"
Jul 23 18:13:07.493905 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.493877    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f"} err="failed to get container status \"da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f\": rpc error: code = NotFound desc = could not find container \"da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f\": container with ID starting with da01b2601a9d16e811cffb9922d7229d11d1dbb80d54d23944d9a9faf082696f not found: ID does not exist"
Jul 23 18:13:07.509646 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.509621    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d5c65d7b9-9vpdh"]
Jul 23 18:13:07.523042 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.523015    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d5c65d7b9-9vpdh"]
Jul 23 18:13:07.608796 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:07.608760    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5f8707-671a-4fe8-b1cd-25e11cea4073" path="/var/lib/kubelet/pods/fa5f8707-671a-4fe8-b1cd-25e11cea4073/volumes"
Jul 23 18:13:53.830084 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:53.830044    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-jnjtl"]
Jul 23 18:13:53.830788 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:53.830256    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl" podUID="cd1f044f-a652-4d35-8df7-07f5bf87f45b" containerName="manager" containerID="cri-o://0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3" gracePeriod=30
Jul 23 18:13:54.082602 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.082490    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:13:54.187185 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.187150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1f044f-a652-4d35-8df7-07f5bf87f45b-webhook-cert\") pod \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") "
Jul 23 18:13:54.187357 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.187206    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfxrq\" (UniqueName: \"kubernetes.io/projected/cd1f044f-a652-4d35-8df7-07f5bf87f45b-kube-api-access-tfxrq\") pod \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\" (UID: \"cd1f044f-a652-4d35-8df7-07f5bf87f45b\") "
Jul 23 18:13:54.189418 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.189388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1f044f-a652-4d35-8df7-07f5bf87f45b-kube-api-access-tfxrq" (OuterVolumeSpecName: "kube-api-access-tfxrq") pod "cd1f044f-a652-4d35-8df7-07f5bf87f45b" (UID: "cd1f044f-a652-4d35-8df7-07f5bf87f45b"). InnerVolumeSpecName "kube-api-access-tfxrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:13:54.189418 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.189389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1f044f-a652-4d35-8df7-07f5bf87f45b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cd1f044f-a652-4d35-8df7-07f5bf87f45b" (UID: "cd1f044f-a652-4d35-8df7-07f5bf87f45b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:13:54.288636 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.288592    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1f044f-a652-4d35-8df7-07f5bf87f45b-webhook-cert\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:13:54.288636 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.288632    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxrq\" (UniqueName: \"kubernetes.io/projected/cd1f044f-a652-4d35-8df7-07f5bf87f45b-kube-api-access-tfxrq\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:13:54.727293 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.727261    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd1f044f-a652-4d35-8df7-07f5bf87f45b" containerID="0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3" exitCode=0
Jul 23 18:13:54.727477 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.727311    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl"
Jul 23 18:13:54.727477 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.727347    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl" event={"ID":"cd1f044f-a652-4d35-8df7-07f5bf87f45b","Type":"ContainerDied","Data":"0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3"}
Jul 23 18:13:54.727477 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.727404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-jnjtl" event={"ID":"cd1f044f-a652-4d35-8df7-07f5bf87f45b","Type":"ContainerDied","Data":"a61b1031e1232355de2003a88f78c52a31fc7c748a43aae94dd905b6947592b4"}
Jul 23 18:13:54.727477 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.727433    2562 scope.go:117] "RemoveContainer" containerID="0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3"
Jul 23 18:13:54.736723 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.736706    2562 scope.go:117] "RemoveContainer" containerID="0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3"
Jul 23 18:13:54.736996 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:13:54.736976    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3\": container with ID starting with 0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3 not found: ID does not exist" containerID="0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3"
Jul 23 18:13:54.737060 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.737005    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3"} err="failed to get container status \"0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3\": rpc error: code = NotFound desc = could not find container \"0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3\": container with ID starting with 0d0ad96c4d5684d60a48a1b581668a2a65af186637d5e90f52129efd69e328d3 not found: ID does not exist"
Jul 23 18:13:54.754593 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.754566    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-jnjtl"]
Jul 23 18:13:54.760415 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:54.760393    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-jnjtl"]
Jul 23 18:13:55.608662 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:13:55.608627    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1f044f-a652-4d35-8df7-07f5bf87f45b" path="/var/lib/kubelet/pods/cd1f044f-a652-4d35-8df7-07f5bf87f45b/volumes"
Jul 23 18:14:21.585407 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:21.585322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:14:21.589247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:21.589211    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:14:55.660013 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.659976    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-5wq4d"]
Jul 23 18:14:55.660517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.660362    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1f044f-a652-4d35-8df7-07f5bf87f45b" containerName="manager"
Jul 23 18:14:55.660517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.660373    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1f044f-a652-4d35-8df7-07f5bf87f45b" containerName="manager"
Jul 23 18:14:55.660517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.660387    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5f8707-671a-4fe8-b1cd-25e11cea4073" containerName="authorino"
Jul 23 18:14:55.660517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.660392    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5f8707-671a-4fe8-b1cd-25e11cea4073" containerName="authorino"
Jul 23 18:14:55.660517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.660444    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1f044f-a652-4d35-8df7-07f5bf87f45b" containerName="manager"
Jul 23 18:14:55.660517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.660453    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5f8707-671a-4fe8-b1cd-25e11cea4073" containerName="authorino"
Jul 23 18:14:55.663414 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.663397    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.666497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.666474    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p7688\""
Jul 23 18:14:55.666497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.666492    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 18:14:55.666680 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.666478    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 18:14:55.673396 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.673370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-5wq4d"]
Jul 23 18:14:55.826723 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.826687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9797w\" (UniqueName: \"kubernetes.io/projected/ec2f803f-812e-456a-91f7-e387142a6839-kube-api-access-9797w\") pod \"maas-controller-86bcf48f9b-5wq4d\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") " pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.826887 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.826793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec2f803f-812e-456a-91f7-e387142a6839-webhook-cert\") pod \"maas-controller-86bcf48f9b-5wq4d\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") " pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.928148 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.928066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9797w\" (UniqueName: \"kubernetes.io/projected/ec2f803f-812e-456a-91f7-e387142a6839-kube-api-access-9797w\") pod \"maas-controller-86bcf48f9b-5wq4d\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") " pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.928148 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.928121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec2f803f-812e-456a-91f7-e387142a6839-webhook-cert\") pod \"maas-controller-86bcf48f9b-5wq4d\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") " pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.930621 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.930598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec2f803f-812e-456a-91f7-e387142a6839-webhook-cert\") pod \"maas-controller-86bcf48f9b-5wq4d\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") " pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.937079 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.937055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9797w\" (UniqueName: \"kubernetes.io/projected/ec2f803f-812e-456a-91f7-e387142a6839-kube-api-access-9797w\") pod \"maas-controller-86bcf48f9b-5wq4d\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") " pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:55.975100 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:55.975063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:56.110301 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:56.110272    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-5wq4d"]
Jul 23 18:14:56.112360 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:14:56.112329    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2f803f_812e_456a_91f7_e387142a6839.slice/crio-99b5ae6960ed3533c3df9ee388e7c210a1b00d493c732b00446b4b6f14818585 WatchSource:0}: Error finding container 99b5ae6960ed3533c3df9ee388e7c210a1b00d493c732b00446b4b6f14818585: Status 404 returned error can't find the container with id 99b5ae6960ed3533c3df9ee388e7c210a1b00d493c732b00446b4b6f14818585
Jul 23 18:14:57.055350 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:57.055314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d" event={"ID":"ec2f803f-812e-456a-91f7-e387142a6839","Type":"ContainerStarted","Data":"8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7"}
Jul 23 18:14:57.055734 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:57.055365    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d" event={"ID":"ec2f803f-812e-456a-91f7-e387142a6839","Type":"ContainerStarted","Data":"99b5ae6960ed3533c3df9ee388e7c210a1b00d493c732b00446b4b6f14818585"}
Jul 23 18:14:57.055734 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:57.055410    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:14:57.078509 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:14:57.078439    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d" podStartSLOduration=1.4846420980000001 podStartE2EDuration="2.078425281s" totalImagesPullingTime="593.783183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:14:57.076130066 +0000 UTC m=+935.866287185" watchObservedRunningTime="2026-07-23 18:14:57.078425281 +0000 UTC m=+935.868582396"
Jul 23 18:15:00.163277 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:00.163246    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"]
Jul 23 18:15:00.166650 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:00.166630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"
Jul 23 18:15:00.169570 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:00.169550    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:15:00.175362 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:00.175328    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"]
Jul 23 18:15:00.178289 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:00.178270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"
Jul 23 18:15:00.316724 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:00.316693    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"]
Jul 23 18:15:00.318094 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:15:00.318061    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e5af57_2870_4ab5_a140_3f0082ff7cc0.slice/crio-c23072cfa6b07bfd2b6b1655eb9e56c1c84ef6138d20fa6292d77bc9096afcf0 WatchSource:0}: Error finding container c23072cfa6b07bfd2b6b1655eb9e56c1c84ef6138d20fa6292d77bc9096afcf0: Status 404 returned error can't find the container with id c23072cfa6b07bfd2b6b1655eb9e56c1c84ef6138d20fa6292d77bc9096afcf0
Jul 23 18:15:01.077103 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:01.077068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g" event={"ID":"78e5af57-2870-4ab5-a140-3f0082ff7cc0","Type":"ContainerStarted","Data":"c23072cfa6b07bfd2b6b1655eb9e56c1c84ef6138d20fa6292d77bc9096afcf0"}
Jul 23 18:15:04.097562 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:04.097520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g" event={"ID":"78e5af57-2870-4ab5-a140-3f0082ff7cc0","Type":"ContainerStarted","Data":"a86dae28f9bc67aaf2c86beca917861b7ae7daeca38fea94916f274129e20833"}
Jul 23 18:15:04.115236 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:04.115188    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g" podStartSLOduration=0.841624461 podStartE2EDuration="4.115173367s" totalImagesPullingTime="3.273548906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:15:04.11340563 +0000 UTC m=+942.903562756" watchObservedRunningTime="2026-07-23 18:15:04.115173367 +0000 UTC m=+942.905330485"
Jul 23 18:15:05.104660 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:05.104623    2562 generic.go:356] "Generic (PLEG): container finished" podID="78e5af57-2870-4ab5-a140-3f0082ff7cc0" containerID="a86dae28f9bc67aaf2c86beca917861b7ae7daeca38fea94916f274129e20833" exitCode=0
Jul 23 18:15:05.105069 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:05.104708    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g" event={"ID":"78e5af57-2870-4ab5-a140-3f0082ff7cc0","Type":"ContainerDied","Data":"a86dae28f9bc67aaf2c86beca917861b7ae7daeca38fea94916f274129e20833"}
Jul 23 18:15:06.258354 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:06.258330    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"
Jul 23 18:15:07.116436 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:07.116401    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g"
Jul 23 18:15:07.116436 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:07.116415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-pnc6g" event={"ID":"78e5af57-2870-4ab5-a140-3f0082ff7cc0","Type":"ContainerDied","Data":"c23072cfa6b07bfd2b6b1655eb9e56c1c84ef6138d20fa6292d77bc9096afcf0"}
Jul 23 18:15:07.116648 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:07.116463    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c23072cfa6b07bfd2b6b1655eb9e56c1c84ef6138d20fa6292d77bc9096afcf0"
Jul 23 18:15:08.062591 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:08.062558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:15:12.774889 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:12.774856    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-5wq4d"]
Jul 23 18:15:12.777432 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:12.775108    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d" podUID="ec2f803f-812e-456a-91f7-e387142a6839" containerName="manager" containerID="cri-o://8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7" gracePeriod=30
Jul 23 18:15:13.123016 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.122993    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:15:13.148532 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.148503    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec2f803f-812e-456a-91f7-e387142a6839" containerID="8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7" exitCode=0
Jul 23 18:15:13.148689 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.148542    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d"
Jul 23 18:15:13.148689 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.148581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d" event={"ID":"ec2f803f-812e-456a-91f7-e387142a6839","Type":"ContainerDied","Data":"8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7"}
Jul 23 18:15:13.148689 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.148631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-5wq4d" event={"ID":"ec2f803f-812e-456a-91f7-e387142a6839","Type":"ContainerDied","Data":"99b5ae6960ed3533c3df9ee388e7c210a1b00d493c732b00446b4b6f14818585"}
Jul 23 18:15:13.148689 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.148653    2562 scope.go:117] "RemoveContainer" containerID="8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7"
Jul 23 18:15:13.157242 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.157225    2562 scope.go:117] "RemoveContainer" containerID="8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7"
Jul 23 18:15:13.157513 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:15:13.157493    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7\": container with ID starting with 8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7 not found: ID does not exist" containerID="8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7"
Jul 23 18:15:13.157607 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.157521    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7"} err="failed to get container status \"8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7\": rpc error: code = NotFound desc = could not find container \"8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7\": container with ID starting with 8c4683a81f63c1ff133a63c6ea0e8bd0ae468d65e886e37a967655f267bb3dd7 not found: ID does not exist"
Jul 23 18:15:13.159839 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.159821    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9797w\" (UniqueName: \"kubernetes.io/projected/ec2f803f-812e-456a-91f7-e387142a6839-kube-api-access-9797w\") pod \"ec2f803f-812e-456a-91f7-e387142a6839\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") "
Jul 23 18:15:13.159897 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.159889    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec2f803f-812e-456a-91f7-e387142a6839-webhook-cert\") pod \"ec2f803f-812e-456a-91f7-e387142a6839\" (UID: \"ec2f803f-812e-456a-91f7-e387142a6839\") "
Jul 23 18:15:13.161902 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.161878    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2f803f-812e-456a-91f7-e387142a6839-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ec2f803f-812e-456a-91f7-e387142a6839" (UID: "ec2f803f-812e-456a-91f7-e387142a6839"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:15:13.162022 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.161970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2f803f-812e-456a-91f7-e387142a6839-kube-api-access-9797w" (OuterVolumeSpecName: "kube-api-access-9797w") pod "ec2f803f-812e-456a-91f7-e387142a6839" (UID: "ec2f803f-812e-456a-91f7-e387142a6839"). InnerVolumeSpecName "kube-api-access-9797w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:15:13.260781 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.260742    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec2f803f-812e-456a-91f7-e387142a6839-webhook-cert\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:15:13.260781 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.260774    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9797w\" (UniqueName: \"kubernetes.io/projected/ec2f803f-812e-456a-91f7-e387142a6839-kube-api-access-9797w\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:15:13.473590 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.473552    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-5wq4d"]
Jul 23 18:15:13.476633 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.476610    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-5wq4d"]
Jul 23 18:15:13.609901 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:13.609858    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2f803f-812e-456a-91f7-e387142a6839" path="/var/lib/kubelet/pods/ec2f803f-812e-456a-91f7-e387142a6839/volumes"
Jul 23 18:15:14.355575 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.355543    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-ldsm9"]
Jul 23 18:15:14.356194 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.356171    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2f803f-812e-456a-91f7-e387142a6839" containerName="manager"
Jul 23 18:15:14.356194 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.356195    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2f803f-812e-456a-91f7-e387142a6839" containerName="manager"
Jul 23 18:15:14.356456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.356215    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e5af57-2870-4ab5-a140-3f0082ff7cc0" containerName="cleanup"
Jul 23 18:15:14.356456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.356225    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e5af57-2870-4ab5-a140-3f0082ff7cc0" containerName="cleanup"
Jul 23 18:15:14.356456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.356309    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2f803f-812e-456a-91f7-e387142a6839" containerName="manager"
Jul 23 18:15:14.356456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.356321    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e5af57-2870-4ab5-a140-3f0082ff7cc0" containerName="cleanup"
Jul 23 18:15:14.360920 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.360902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.364532 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.364509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p7688\""
Jul 23 18:15:14.364693 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.364553    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 18:15:14.364693 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.364509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 18:15:14.366877 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.366852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91e7e0bb-57d8-4d84-976e-8f92f2f761db-webhook-cert\") pod \"maas-controller-86bcf48f9b-ldsm9\" (UID: \"91e7e0bb-57d8-4d84-976e-8f92f2f761db\") " pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.367017 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.366881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpm7\" (UniqueName: \"kubernetes.io/projected/91e7e0bb-57d8-4d84-976e-8f92f2f761db-kube-api-access-hgpm7\") pod \"maas-controller-86bcf48f9b-ldsm9\" (UID: \"91e7e0bb-57d8-4d84-976e-8f92f2f761db\") " pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.371659 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.371635    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-ldsm9"]
Jul 23 18:15:14.468102 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.468064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91e7e0bb-57d8-4d84-976e-8f92f2f761db-webhook-cert\") pod \"maas-controller-86bcf48f9b-ldsm9\" (UID: \"91e7e0bb-57d8-4d84-976e-8f92f2f761db\") " pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.468102 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.468101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpm7\" (UniqueName: \"kubernetes.io/projected/91e7e0bb-57d8-4d84-976e-8f92f2f761db-kube-api-access-hgpm7\") pod \"maas-controller-86bcf48f9b-ldsm9\" (UID: \"91e7e0bb-57d8-4d84-976e-8f92f2f761db\") " pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.470865 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.470834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91e7e0bb-57d8-4d84-976e-8f92f2f761db-webhook-cert\") pod \"maas-controller-86bcf48f9b-ldsm9\" (UID: \"91e7e0bb-57d8-4d84-976e-8f92f2f761db\") " pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.477094 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.477055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpm7\" (UniqueName: \"kubernetes.io/projected/91e7e0bb-57d8-4d84-976e-8f92f2f761db-kube-api-access-hgpm7\") pod \"maas-controller-86bcf48f9b-ldsm9\" (UID: \"91e7e0bb-57d8-4d84-976e-8f92f2f761db\") " pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.674005 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.673895    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:14.808313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:14.808278    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf48f9b-ldsm9"]
Jul 23 18:15:14.809973 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:15:14.809930    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e7e0bb_57d8_4d84_976e_8f92f2f761db.slice/crio-dc7976fea96cae03bbb6f2db8d13248d06584b6b20967714b82877701a56a1cd WatchSource:0}: Error finding container dc7976fea96cae03bbb6f2db8d13248d06584b6b20967714b82877701a56a1cd: Status 404 returned error can't find the container with id dc7976fea96cae03bbb6f2db8d13248d06584b6b20967714b82877701a56a1cd
Jul 23 18:15:15.161800 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:15.161764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9" event={"ID":"91e7e0bb-57d8-4d84-976e-8f92f2f761db","Type":"ContainerStarted","Data":"dc7976fea96cae03bbb6f2db8d13248d06584b6b20967714b82877701a56a1cd"}
Jul 23 18:15:16.167750 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:16.167716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9" event={"ID":"91e7e0bb-57d8-4d84-976e-8f92f2f761db","Type":"ContainerStarted","Data":"2af1b103238c8c1941bd5d0a2e19355687a20a86e8db78ad451e1dbadf98c9b0"}
Jul 23 18:15:16.168272 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:16.167850    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:15:16.186265 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:16.186212    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9" podStartSLOduration=1.7320662919999998 podStartE2EDuration="2.186198048s" totalImagesPullingTime="454.131756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:15:16.184790366 +0000 UTC m=+954.974947493" watchObservedRunningTime="2026-07-23 18:15:16.186198048 +0000 UTC m=+954.976355162"
Jul 23 18:15:27.176700 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:15:27.176664    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86bcf48f9b-ldsm9"
Jul 23 18:19:21.627862 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:19:21.627826    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:19:21.633348 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:19:21.633328    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:20:07.593139 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:07.593105    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"]
Jul 23 18:20:07.593717 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:07.593296    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m" podUID="268e8bd4-e7f4-44a7-8b51-d9ccf7040535" containerName="manager" containerID="cri-o://98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4" gracePeriod=10
Jul 23 18:20:10.351056 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.351027    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:20:10.465376 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.465287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-extensions-socket-volume\") pod \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") "
Jul 23 18:20:10.465376 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.465374    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8669\" (UniqueName: \"kubernetes.io/projected/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-kube-api-access-j8669\") pod \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\" (UID: \"268e8bd4-e7f4-44a7-8b51-d9ccf7040535\") "
Jul 23 18:20:10.466097 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.465749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "268e8bd4-e7f4-44a7-8b51-d9ccf7040535" (UID: "268e8bd4-e7f4-44a7-8b51-d9ccf7040535"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 18:20:10.468590 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.468477    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-kube-api-access-j8669" (OuterVolumeSpecName: "kube-api-access-j8669") pod "268e8bd4-e7f4-44a7-8b51-d9ccf7040535" (UID: "268e8bd4-e7f4-44a7-8b51-d9ccf7040535"). InnerVolumeSpecName "kube-api-access-j8669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:20:10.566087 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.566043    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-extensions-socket-volume\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:20:10.566087 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.566079    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8669\" (UniqueName: \"kubernetes.io/projected/268e8bd4-e7f4-44a7-8b51-d9ccf7040535-kube-api-access-j8669\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:20:10.734486 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.734397    2562 generic.go:356] "Generic (PLEG): container finished" podID="268e8bd4-e7f4-44a7-8b51-d9ccf7040535" containerID="98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4" exitCode=0
Jul 23 18:20:10.734486 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.734461    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"
Jul 23 18:20:10.734793 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.734460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m" event={"ID":"268e8bd4-e7f4-44a7-8b51-d9ccf7040535","Type":"ContainerDied","Data":"98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4"}
Jul 23 18:20:10.734793 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.734585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m" event={"ID":"268e8bd4-e7f4-44a7-8b51-d9ccf7040535","Type":"ContainerDied","Data":"1d60a8abe943af1836247cacda8c5d2accf7fc4e7110128c7a783ff4008d1bf5"}
Jul 23 18:20:10.734793 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.734608    2562 scope.go:117] "RemoveContainer" containerID="98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4"
Jul 23 18:20:10.745109 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.745092    2562 scope.go:117] "RemoveContainer" containerID="98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4"
Jul 23 18:20:10.745340 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:20:10.745320    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4\": container with ID starting with 98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4 not found: ID does not exist" containerID="98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4"
Jul 23 18:20:10.745404 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.745348    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4"} err="failed to get container status \"98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4\": rpc error: code = NotFound desc = could not find container \"98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4\": container with ID starting with 98d3fced9b5a291b973dcd70ad45cacf578a85ced8eb0edf60bf1307ca78c6d4 not found: ID does not exist"
Jul 23 18:20:10.755819 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.755798    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"]
Jul 23 18:20:10.759497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:10.759477    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-64z9m"]
Jul 23 18:20:11.608032 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:20:11.607996    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268e8bd4-e7f4-44a7-8b51-d9ccf7040535" path="/var/lib/kubelet/pods/268e8bd4-e7f4-44a7-8b51-d9ccf7040535/volumes"
Jul 23 18:21:08.753554 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.753510    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"]
Jul 23 18:21:08.754384 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.754356    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268e8bd4-e7f4-44a7-8b51-d9ccf7040535" containerName="manager"
Jul 23 18:21:08.754509 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.754388    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="268e8bd4-e7f4-44a7-8b51-d9ccf7040535" containerName="manager"
Jul 23 18:21:08.754605 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.754590    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="268e8bd4-e7f4-44a7-8b51-d9ccf7040535" containerName="manager"
Jul 23 18:21:08.758137 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.758116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:08.760317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.760294    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ppxlz\""
Jul 23 18:21:08.768127 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.768097    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"]
Jul 23 18:21:08.865063 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.865037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jdj\" (UniqueName: \"kubernetes.io/projected/1853da34-be56-4fbf-bcf5-8c1a0231e6fe-kube-api-access-c2jdj\") pod \"kuadrant-operator-controller-manager-86878c769f-brq6n\" (UID: \"1853da34-be56-4fbf-bcf5-8c1a0231e6fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:08.865226 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.865071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1853da34-be56-4fbf-bcf5-8c1a0231e6fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-brq6n\" (UID: \"1853da34-be56-4fbf-bcf5-8c1a0231e6fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:08.965712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.965676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jdj\" (UniqueName: \"kubernetes.io/projected/1853da34-be56-4fbf-bcf5-8c1a0231e6fe-kube-api-access-c2jdj\") pod \"kuadrant-operator-controller-manager-86878c769f-brq6n\" (UID: \"1853da34-be56-4fbf-bcf5-8c1a0231e6fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:08.965712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.965716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1853da34-be56-4fbf-bcf5-8c1a0231e6fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-brq6n\" (UID: \"1853da34-be56-4fbf-bcf5-8c1a0231e6fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:08.966106 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.966086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1853da34-be56-4fbf-bcf5-8c1a0231e6fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-brq6n\" (UID: \"1853da34-be56-4fbf-bcf5-8c1a0231e6fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:08.981564 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:08.981530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jdj\" (UniqueName: \"kubernetes.io/projected/1853da34-be56-4fbf-bcf5-8c1a0231e6fe-kube-api-access-c2jdj\") pod \"kuadrant-operator-controller-manager-86878c769f-brq6n\" (UID: \"1853da34-be56-4fbf-bcf5-8c1a0231e6fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:09.069139 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:09.069059    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:09.215479 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:09.215368    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"]
Jul 23 18:21:09.218462 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:21:09.218435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1853da34_be56_4fbf_bcf5_8c1a0231e6fe.slice/crio-ccc8248575bf14c025f86f183d72798962f6a1508d71b2b6612fa60b4098b116 WatchSource:0}: Error finding container ccc8248575bf14c025f86f183d72798962f6a1508d71b2b6612fa60b4098b116: Status 404 returned error can't find the container with id ccc8248575bf14c025f86f183d72798962f6a1508d71b2b6612fa60b4098b116
Jul 23 18:21:10.034200 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:10.034157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n" event={"ID":"1853da34-be56-4fbf-bcf5-8c1a0231e6fe","Type":"ContainerStarted","Data":"356c0d47d892e298ddfc8caa205c2e538f7013f3982fb4405407350212a04401"}
Jul 23 18:21:10.034653 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:10.034301    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:21:10.034653 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:10.034319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n" event={"ID":"1853da34-be56-4fbf-bcf5-8c1a0231e6fe","Type":"ContainerStarted","Data":"ccc8248575bf14c025f86f183d72798962f6a1508d71b2b6612fa60b4098b116"}
Jul 23 18:21:10.063139 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:10.063090    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n" podStartSLOduration=2.063074227 podStartE2EDuration="2.063074227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:10.061967868 +0000 UTC m=+1308.852124980" watchObservedRunningTime="2026-07-23 18:21:10.063074227 +0000 UTC m=+1308.853231341"
Jul 23 18:21:21.041902 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:21:21.041872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-brq6n"
Jul 23 18:24:21.661875 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:24:21.661835    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:24:21.668232 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:24:21.668206    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:28:14.740008 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.739968    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"]
Jul 23 18:28:14.743745 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.743729    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.746651 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.746624    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:28:14.747221 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.747207    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0d2d0218-serving-cert\""
Jul 23 18:28:14.765972 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.765933    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"]
Jul 23 18:28:14.887343 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.887301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.887533 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.887389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szv79\" (UniqueName: \"kubernetes.io/projected/7556f284-2098-4ccf-90c0-2ff3ef7b873a-kube-api-access-szv79\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.887533 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.887451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7556f284-2098-4ccf-90c0-2ff3ef7b873a-openshift-service-ca\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.988295 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.988244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szv79\" (UniqueName: \"kubernetes.io/projected/7556f284-2098-4ccf-90c0-2ff3ef7b873a-kube-api-access-szv79\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.988492 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.988362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7556f284-2098-4ccf-90c0-2ff3ef7b873a-openshift-service-ca\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.988492 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.988405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:14.988566 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:28:14.988534    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-serving-cert: secret "maas-api-e2e-ait-0d2d0218-serving-cert" not found
Jul 23 18:28:14.988608 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:28:14.988597    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls podName:7556f284-2098-4ccf-90c0-2ff3ef7b873a nodeName:}" failed. No retries permitted until 2026-07-23 18:28:15.488577208 +0000 UTC m=+1734.278734303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls") pod "maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" (UID: "7556f284-2098-4ccf-90c0-2ff3ef7b873a") : secret "maas-api-e2e-ait-0d2d0218-serving-cert" not found
Jul 23 18:28:14.989008 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:14.988990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7556f284-2098-4ccf-90c0-2ff3ef7b873a-openshift-service-ca\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:15.000351 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:15.000284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szv79\" (UniqueName: \"kubernetes.io/projected/7556f284-2098-4ccf-90c0-2ff3ef7b873a-kube-api-access-szv79\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:15.492658 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:15.492615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:15.495404 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:15.495347    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls\") pod \"maas-api-e2e-ait-0d2d0218-67465f6769-5crqr\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:15.653770 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:15.653742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:15.800173 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:15.800138    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"]
Jul 23 18:28:15.801346 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:28:15.801320    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7556f284_2098_4ccf_90c0_2ff3ef7b873a.slice/crio-5be87997fdd168d0706ba22808df0db1fbbb388b7af9346f664ff904e65f1b4a WatchSource:0}: Error finding container 5be87997fdd168d0706ba22808df0db1fbbb388b7af9346f664ff904e65f1b4a: Status 404 returned error can't find the container with id 5be87997fdd168d0706ba22808df0db1fbbb388b7af9346f664ff904e65f1b4a
Jul 23 18:28:15.802850 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:15.802832    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:28:16.305251 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:16.305215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" event={"ID":"7556f284-2098-4ccf-90c0-2ff3ef7b873a","Type":"ContainerStarted","Data":"5be87997fdd168d0706ba22808df0db1fbbb388b7af9346f664ff904e65f1b4a"}
Jul 23 18:28:17.312164 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:17.312127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" event={"ID":"7556f284-2098-4ccf-90c0-2ff3ef7b873a","Type":"ContainerStarted","Data":"16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67"}
Jul 23 18:28:17.312531 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:17.312257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:17.332579 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:17.332528    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" podStartSLOduration=1.896648533 podStartE2EDuration="3.332508374s" totalImagesPullingTime="1.435859841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:28:17.331501328 +0000 UTC m=+1736.121658442" watchObservedRunningTime="2026-07-23 18:28:17.332508374 +0000 UTC m=+1736.122665510"
Jul 23 18:28:23.322762 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:23.322730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:40.074188 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.074148    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"]
Jul 23 18:28:40.078149 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.078129    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:40.080528 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.080505    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:28:40.084839 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.084815    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"]
Jul 23 18:28:40.092211 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.092177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab35efe-44d1-4a4b-9bc5-8e15cae16adc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn\" (UID: \"cab35efe-44d1-4a4b-9bc5-8e15cae16adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:40.193608 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.193565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab35efe-44d1-4a4b-9bc5-8e15cae16adc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn\" (UID: \"cab35efe-44d1-4a4b-9bc5-8e15cae16adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:40.194264 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.194240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab35efe-44d1-4a4b-9bc5-8e15cae16adc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn\" (UID: \"cab35efe-44d1-4a4b-9bc5-8e15cae16adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:40.391078 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.390994    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:40.534869 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:40.534837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"]
Jul 23 18:28:40.536293 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:28:40.536267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab35efe_44d1_4a4b_9bc5_8e15cae16adc.slice/crio-a592e1f8651b6436c7d8cba3fc571b5995661fdf74ed4c738de09df4e11115ef WatchSource:0}: Error finding container a592e1f8651b6436c7d8cba3fc571b5995661fdf74ed4c738de09df4e11115ef: Status 404 returned error can't find the container with id a592e1f8651b6436c7d8cba3fc571b5995661fdf74ed4c738de09df4e11115ef
Jul 23 18:28:41.442397 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:41.442361    2562 generic.go:356] "Generic (PLEG): container finished" podID="cab35efe-44d1-4a4b-9bc5-8e15cae16adc" containerID="0b0a381c11bcda1d14e3dec1104b1773e8d05b226ab436033f91be3e256c2531" exitCode=0
Jul 23 18:28:41.442804 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:41.442436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn" event={"ID":"cab35efe-44d1-4a4b-9bc5-8e15cae16adc","Type":"ContainerDied","Data":"0b0a381c11bcda1d14e3dec1104b1773e8d05b226ab436033f91be3e256c2531"}
Jul 23 18:28:41.442804 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:41.442497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn" event={"ID":"cab35efe-44d1-4a4b-9bc5-8e15cae16adc","Type":"ContainerStarted","Data":"a592e1f8651b6436c7d8cba3fc571b5995661fdf74ed4c738de09df4e11115ef"}
Jul 23 18:28:42.581913 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:42.581886    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:42.617315 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:42.617287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab35efe-44d1-4a4b-9bc5-8e15cae16adc-maas-api-service-ca\") pod \"cab35efe-44d1-4a4b-9bc5-8e15cae16adc\" (UID: \"cab35efe-44d1-4a4b-9bc5-8e15cae16adc\") "
Jul 23 18:28:42.617642 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:42.617615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab35efe-44d1-4a4b-9bc5-8e15cae16adc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cab35efe-44d1-4a4b-9bc5-8e15cae16adc" (UID: "cab35efe-44d1-4a4b-9bc5-8e15cae16adc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:28:42.718307 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:42.718219    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab35efe-44d1-4a4b-9bc5-8e15cae16adc-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:28:43.454222 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:43.454193    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"
Jul 23 18:28:43.454421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:43.454235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn" event={"ID":"cab35efe-44d1-4a4b-9bc5-8e15cae16adc","Type":"ContainerDied","Data":"a592e1f8651b6436c7d8cba3fc571b5995661fdf74ed4c738de09df4e11115ef"}
Jul 23 18:28:43.454421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:43.454283    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a592e1f8651b6436c7d8cba3fc571b5995661fdf74ed4c738de09df4e11115ef"
Jul 23 18:28:50.614957 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.614905    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"]
Jul 23 18:28:50.615450 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.615151    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" podUID="7556f284-2098-4ccf-90c0-2ff3ef7b873a" containerName="maas-api" containerID="cri-o://16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67" gracePeriod=30
Jul 23 18:28:50.866187 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.866118    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:50.992180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.992146    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7556f284-2098-4ccf-90c0-2ff3ef7b873a-openshift-service-ca\") pod \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") "
Jul 23 18:28:50.992180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.992187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls\") pod \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") "
Jul 23 18:28:50.992419 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.992231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szv79\" (UniqueName: \"kubernetes.io/projected/7556f284-2098-4ccf-90c0-2ff3ef7b873a-kube-api-access-szv79\") pod \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\" (UID: \"7556f284-2098-4ccf-90c0-2ff3ef7b873a\") "
Jul 23 18:28:50.992499 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.992476    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7556f284-2098-4ccf-90c0-2ff3ef7b873a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7556f284-2098-4ccf-90c0-2ff3ef7b873a" (UID: "7556f284-2098-4ccf-90c0-2ff3ef7b873a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:28:50.994646 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.994603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7556f284-2098-4ccf-90c0-2ff3ef7b873a-kube-api-access-szv79" (OuterVolumeSpecName: "kube-api-access-szv79") pod "7556f284-2098-4ccf-90c0-2ff3ef7b873a" (UID: "7556f284-2098-4ccf-90c0-2ff3ef7b873a"). InnerVolumeSpecName "kube-api-access-szv79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:28:50.994646 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:50.994612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7556f284-2098-4ccf-90c0-2ff3ef7b873a" (UID: "7556f284-2098-4ccf-90c0-2ff3ef7b873a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:28:51.093263 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.093225    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7556f284-2098-4ccf-90c0-2ff3ef7b873a-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:28:51.093263 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.093257    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7556f284-2098-4ccf-90c0-2ff3ef7b873a-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:28:51.093263 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.093266    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szv79\" (UniqueName: \"kubernetes.io/projected/7556f284-2098-4ccf-90c0-2ff3ef7b873a-kube-api-access-szv79\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:28:51.498189 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.498155    2562 generic.go:356] "Generic (PLEG): container finished" podID="7556f284-2098-4ccf-90c0-2ff3ef7b873a" containerID="16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67" exitCode=0
Jul 23 18:28:51.498379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.498212    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"
Jul 23 18:28:51.498379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.498242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" event={"ID":"7556f284-2098-4ccf-90c0-2ff3ef7b873a","Type":"ContainerDied","Data":"16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67"}
Jul 23 18:28:51.498379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.498306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr" event={"ID":"7556f284-2098-4ccf-90c0-2ff3ef7b873a","Type":"ContainerDied","Data":"5be87997fdd168d0706ba22808df0db1fbbb388b7af9346f664ff904e65f1b4a"}
Jul 23 18:28:51.498379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.498336    2562 scope.go:117] "RemoveContainer" containerID="16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67"
Jul 23 18:28:51.508194 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.508181    2562 scope.go:117] "RemoveContainer" containerID="16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67"
Jul 23 18:28:51.508474 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:28:51.508458    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67\": container with ID starting with 16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67 not found: ID does not exist" containerID="16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67"
Jul 23 18:28:51.508513 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.508483    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67"} err="failed to get container status \"16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67\": rpc error: code = NotFound desc = could not find container \"16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67\": container with ID starting with 16827774f495fe7c4759f6c429b86642546cbb2263b39d9227cc874a7012ad67 not found: ID does not exist"
Jul 23 18:28:51.521306 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.521277    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"]
Jul 23 18:28:51.525330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.525307    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d2d0218-67465f6769-5crqr"]
Jul 23 18:28:51.609000 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:51.608965    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7556f284-2098-4ccf-90c0-2ff3ef7b873a" path="/var/lib/kubelet/pods/7556f284-2098-4ccf-90c0-2ff3ef7b873a/volumes"
Jul 23 18:28:55.146316 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:55.146275    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"]
Jul 23 18:28:55.149969 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:55.149924    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d2d0218-6sjkn"]
Jul 23 18:28:55.607980 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:55.607926    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab35efe-44d1-4a4b-9bc5-8e15cae16adc" path="/var/lib/kubelet/pods/cab35efe-44d1-4a4b-9bc5-8e15cae16adc/volumes"
Jul 23 18:28:58.975183 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975152    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"]
Jul 23 18:28:58.975594 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975562    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab35efe-44d1-4a4b-9bc5-8e15cae16adc" containerName="revoke-keys"
Jul 23 18:28:58.975594 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975575    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab35efe-44d1-4a4b-9bc5-8e15cae16adc" containerName="revoke-keys"
Jul 23 18:28:58.975594 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975587    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7556f284-2098-4ccf-90c0-2ff3ef7b873a" containerName="maas-api"
Jul 23 18:28:58.975594 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975593    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7556f284-2098-4ccf-90c0-2ff3ef7b873a" containerName="maas-api"
Jul 23 18:28:58.975823 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975669    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab35efe-44d1-4a4b-9bc5-8e15cae16adc" containerName="revoke-keys"
Jul 23 18:28:58.975823 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.975681    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7556f284-2098-4ccf-90c0-2ff3ef7b873a" containerName="maas-api"
Jul 23 18:28:58.978979 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.978957    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:58.982652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.982619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:28:58.982882 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.982865    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7b1deb3b-serving-cert\""
Jul 23 18:28:58.991600 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:58.991580    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"]
Jul 23 18:28:59.062969 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.062908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-openshift-service-ca\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.063171 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.063050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-maas-api-tls\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.063171 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.063102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxr7n\" (UniqueName: \"kubernetes.io/projected/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-kube-api-access-hxr7n\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.163954 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.163903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-maas-api-tls\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.164149 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.163974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxr7n\" (UniqueName: \"kubernetes.io/projected/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-kube-api-access-hxr7n\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.164149 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.164019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-openshift-service-ca\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.164718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.164690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-openshift-service-ca\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.166455 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.166438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-maas-api-tls\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.171881 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.171862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxr7n\" (UniqueName: \"kubernetes.io/projected/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-kube-api-access-hxr7n\") pod \"maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.290122 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.290089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:28:59.433509 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.433485    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"]
Jul 23 18:28:59.436270 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:28:59.436241    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d0abdd_cffe_425e_8f5a_e97d0d2e8079.slice/crio-8937d6aa5cb2796266a7e54ad88de187be956265a23adc547d50ae780e142c30 WatchSource:0}: Error finding container 8937d6aa5cb2796266a7e54ad88de187be956265a23adc547d50ae780e142c30: Status 404 returned error can't find the container with id 8937d6aa5cb2796266a7e54ad88de187be956265a23adc547d50ae780e142c30
Jul 23 18:28:59.552026 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:28:59.551905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj" event={"ID":"32d0abdd-cffe-425e-8f5a-e97d0d2e8079","Type":"ContainerStarted","Data":"8937d6aa5cb2796266a7e54ad88de187be956265a23adc547d50ae780e142c30"}
Jul 23 18:29:00.560249 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:00.560217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj" event={"ID":"32d0abdd-cffe-425e-8f5a-e97d0d2e8079","Type":"ContainerStarted","Data":"c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b"}
Jul 23 18:29:00.560698 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:00.560369    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:29:00.578143 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:00.578091    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj" podStartSLOduration=2.274997467 podStartE2EDuration="2.578075922s" totalImagesPullingTime="303.078455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:29:00.576390105 +0000 UTC m=+1779.366547239" watchObservedRunningTime="2026-07-23 18:29:00.578075922 +0000 UTC m=+1779.368233036"
Jul 23 18:29:06.571460 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:06.571426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:29:21.706678 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:21.706648    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:29:21.718086 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:21.718061    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:29:34.478466 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.478428    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"]
Jul 23 18:29:34.479022 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.478760    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj" podUID="32d0abdd-cffe-425e-8f5a-e97d0d2e8079" containerName="maas-api" containerID="cri-o://c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b" gracePeriod=30
Jul 23 18:29:34.729789 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.729722    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:29:34.744254 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.744217    2562 generic.go:356] "Generic (PLEG): container finished" podID="32d0abdd-cffe-425e-8f5a-e97d0d2e8079" containerID="c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b" exitCode=0
Jul 23 18:29:34.744416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.744259    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"
Jul 23 18:29:34.744416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.744307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj" event={"ID":"32d0abdd-cffe-425e-8f5a-e97d0d2e8079","Type":"ContainerDied","Data":"c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b"}
Jul 23 18:29:34.744416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.744357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj" event={"ID":"32d0abdd-cffe-425e-8f5a-e97d0d2e8079","Type":"ContainerDied","Data":"8937d6aa5cb2796266a7e54ad88de187be956265a23adc547d50ae780e142c30"}
Jul 23 18:29:34.744416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.744381    2562 scope.go:117] "RemoveContainer" containerID="c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b"
Jul 23 18:29:34.754838 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.754818    2562 scope.go:117] "RemoveContainer" containerID="c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b"
Jul 23 18:29:34.755251 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:29:34.755230    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b\": container with ID starting with c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b not found: ID does not exist" containerID="c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b"
Jul 23 18:29:34.755337 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.755262    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b"} err="failed to get container status \"c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b\": rpc error: code = NotFound desc = could not find container \"c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b\": container with ID starting with c155e56a0df3c0ce0ddb7682eb9e6e8ef0beffa80ed6a67ec019b4a10900616b not found: ID does not exist"
Jul 23 18:29:34.760584 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.760557    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxr7n\" (UniqueName: \"kubernetes.io/projected/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-kube-api-access-hxr7n\") pod \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") "
Jul 23 18:29:34.760712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.760600    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-openshift-service-ca\") pod \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") "
Jul 23 18:29:34.760712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.760689    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-maas-api-tls\") pod \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\" (UID: \"32d0abdd-cffe-425e-8f5a-e97d0d2e8079\") "
Jul 23 18:29:34.761052 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.761020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32d0abdd-cffe-425e-8f5a-e97d0d2e8079" (UID: "32d0abdd-cffe-425e-8f5a-e97d0d2e8079"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:29:34.762828 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.762804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32d0abdd-cffe-425e-8f5a-e97d0d2e8079" (UID: "32d0abdd-cffe-425e-8f5a-e97d0d2e8079"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:29:34.762913 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.762829    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-kube-api-access-hxr7n" (OuterVolumeSpecName: "kube-api-access-hxr7n") pod "32d0abdd-cffe-425e-8f5a-e97d0d2e8079" (UID: "32d0abdd-cffe-425e-8f5a-e97d0d2e8079"). InnerVolumeSpecName "kube-api-access-hxr7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:29:34.861548 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.861509    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxr7n\" (UniqueName: \"kubernetes.io/projected/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-kube-api-access-hxr7n\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:29:34.861548 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.861546    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:29:34.861548 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:34.861555    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d0abdd-cffe-425e-8f5a-e97d0d2e8079-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:29:35.066927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:35.066878    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"]
Jul 23 18:29:35.071932 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:35.071906    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b1deb3b-5bcbb84789-2dhvj"]
Jul 23 18:29:35.612041 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:35.612002    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d0abdd-cffe-425e-8f5a-e97d0d2e8079" path="/var/lib/kubelet/pods/32d0abdd-cffe-425e-8f5a-e97d0d2e8079/volumes"
Jul 23 18:29:42.072970 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.072917    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"]
Jul 23 18:29:42.073560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.073538    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d0abdd-cffe-425e-8f5a-e97d0d2e8079" containerName="maas-api"
Jul 23 18:29:42.073560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.073563    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d0abdd-cffe-425e-8f5a-e97d0d2e8079" containerName="maas-api"
Jul 23 18:29:42.073709 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.073663    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d0abdd-cffe-425e-8f5a-e97d0d2e8079" containerName="maas-api"
Jul 23 18:29:42.078158 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.078134    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.080758 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.080724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9efa58d0-serving-cert\""
Jul 23 18:29:42.080892 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.080821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:29:42.088716 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.088690    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"]
Jul 23 18:29:42.121479 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.121442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bpj7\" (UniqueName: \"kubernetes.io/projected/8664c542-7802-4307-a3de-d6a78ccf9436-kube-api-access-7bpj7\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.121479 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.121483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8664c542-7802-4307-a3de-d6a78ccf9436-openshift-service-ca\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.121693 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.121546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.222624 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.222580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bpj7\" (UniqueName: \"kubernetes.io/projected/8664c542-7802-4307-a3de-d6a78ccf9436-kube-api-access-7bpj7\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.222817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.222634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8664c542-7802-4307-a3de-d6a78ccf9436-openshift-service-ca\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.222817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.222683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.222906 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:29:42.222831    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-serving-cert: secret "maas-api-e2e-ait-9efa58d0-serving-cert" not found
Jul 23 18:29:42.222975 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:29:42.222908    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls podName:8664c542-7802-4307-a3de-d6a78ccf9436 nodeName:}" failed. No retries permitted until 2026-07-23 18:29:42.722887487 +0000 UTC m=+1821.513044580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls") pod "maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" (UID: "8664c542-7802-4307-a3de-d6a78ccf9436") : secret "maas-api-e2e-ait-9efa58d0-serving-cert" not found
Jul 23 18:29:42.223378 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.223358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8664c542-7802-4307-a3de-d6a78ccf9436-openshift-service-ca\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.234190 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.234165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bpj7\" (UniqueName: \"kubernetes.io/projected/8664c542-7802-4307-a3de-d6a78ccf9436-kube-api-access-7bpj7\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.727280 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.727246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.729833 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.729808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls\") pod \"maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:42.990677 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:42.990582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:43.138461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:43.138433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"]
Jul 23 18:29:43.140036 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:29:43.140001    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8664c542_7802_4307_a3de_d6a78ccf9436.slice/crio-84a7344825d54a66c4a41e306080fabe0ad5363eaff7de6ea1489763c586676c WatchSource:0}: Error finding container 84a7344825d54a66c4a41e306080fabe0ad5363eaff7de6ea1489763c586676c: Status 404 returned error can't find the container with id 84a7344825d54a66c4a41e306080fabe0ad5363eaff7de6ea1489763c586676c
Jul 23 18:29:43.797070 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:43.797031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" event={"ID":"8664c542-7802-4307-a3de-d6a78ccf9436","Type":"ContainerStarted","Data":"32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071"}
Jul 23 18:29:43.797270 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:43.797098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" event={"ID":"8664c542-7802-4307-a3de-d6a78ccf9436","Type":"ContainerStarted","Data":"84a7344825d54a66c4a41e306080fabe0ad5363eaff7de6ea1489763c586676c"}
Jul 23 18:29:43.797656 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:43.797626    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:29:43.814381 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:43.814328    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" podStartSLOduration=1.283876912 podStartE2EDuration="1.814310628s" totalImagesPullingTime="530.433716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:29:43.811966365 +0000 UTC m=+1822.602123478" watchObservedRunningTime="2026-07-23 18:29:43.814310628 +0000 UTC m=+1822.604467750"
Jul 23 18:29:50.816330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:29:50.816298    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:30:18.251476 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.251439    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"]
Jul 23 18:30:18.252011 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.251648    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" podUID="8664c542-7802-4307-a3de-d6a78ccf9436" containerName="maas-api" containerID="cri-o://32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071" gracePeriod=30
Jul 23 18:30:18.512616 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.512592    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:30:18.651814 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.651766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bpj7\" (UniqueName: \"kubernetes.io/projected/8664c542-7802-4307-a3de-d6a78ccf9436-kube-api-access-7bpj7\") pod \"8664c542-7802-4307-a3de-d6a78ccf9436\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") "
Jul 23 18:30:18.651814 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.651822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls\") pod \"8664c542-7802-4307-a3de-d6a78ccf9436\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") "
Jul 23 18:30:18.652077 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.651886    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8664c542-7802-4307-a3de-d6a78ccf9436-openshift-service-ca\") pod \"8664c542-7802-4307-a3de-d6a78ccf9436\" (UID: \"8664c542-7802-4307-a3de-d6a78ccf9436\") "
Jul 23 18:30:18.652293 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.652260    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8664c542-7802-4307-a3de-d6a78ccf9436-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8664c542-7802-4307-a3de-d6a78ccf9436" (UID: "8664c542-7802-4307-a3de-d6a78ccf9436"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:30:18.654128 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.654102    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8664c542-7802-4307-a3de-d6a78ccf9436-kube-api-access-7bpj7" (OuterVolumeSpecName: "kube-api-access-7bpj7") pod "8664c542-7802-4307-a3de-d6a78ccf9436" (UID: "8664c542-7802-4307-a3de-d6a78ccf9436"). InnerVolumeSpecName "kube-api-access-7bpj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:30:18.654217 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.654129    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8664c542-7802-4307-a3de-d6a78ccf9436" (UID: "8664c542-7802-4307-a3de-d6a78ccf9436"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:30:18.753321 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.753287    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bpj7\" (UniqueName: \"kubernetes.io/projected/8664c542-7802-4307-a3de-d6a78ccf9436-kube-api-access-7bpj7\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:30:18.753321 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.753316    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8664c542-7802-4307-a3de-d6a78ccf9436-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:30:18.753321 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.753325    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8664c542-7802-4307-a3de-d6a78ccf9436-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:30:18.992501 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.992409    2562 generic.go:356] "Generic (PLEG): container finished" podID="8664c542-7802-4307-a3de-d6a78ccf9436" containerID="32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071" exitCode=0
Jul 23 18:30:18.992501 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.992451    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"
Jul 23 18:30:18.992728 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.992454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" event={"ID":"8664c542-7802-4307-a3de-d6a78ccf9436","Type":"ContainerDied","Data":"32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071"}
Jul 23 18:30:18.992728 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.992604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz" event={"ID":"8664c542-7802-4307-a3de-d6a78ccf9436","Type":"ContainerDied","Data":"84a7344825d54a66c4a41e306080fabe0ad5363eaff7de6ea1489763c586676c"}
Jul 23 18:30:18.992728 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:18.992640    2562 scope.go:117] "RemoveContainer" containerID="32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071"
Jul 23 18:30:19.002443 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:19.002417    2562 scope.go:117] "RemoveContainer" containerID="32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071"
Jul 23 18:30:19.002742 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:30:19.002721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071\": container with ID starting with 32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071 not found: ID does not exist" containerID="32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071"
Jul 23 18:30:19.002823 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:19.002750    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071"} err="failed to get container status \"32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071\": rpc error: code = NotFound desc = could not find container \"32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071\": container with ID starting with 32aab4c97b9beebb1ef932db1cfe2fd541838eb176c13b78c7edc48cbc1e0071 not found: ID does not exist"
Jul 23 18:30:19.014758 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:19.014724    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"]
Jul 23 18:30:19.020462 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:19.020434    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9efa58d0-5478b7987d-v2vkz"]
Jul 23 18:30:19.615150 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:19.612388    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8664c542-7802-4307-a3de-d6a78ccf9436" path="/var/lib/kubelet/pods/8664c542-7802-4307-a3de-d6a78ccf9436/volumes"
Jul 23 18:30:27.273195 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.273145    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"]
Jul 23 18:30:27.273659 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.273548    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8664c542-7802-4307-a3de-d6a78ccf9436" containerName="maas-api"
Jul 23 18:30:27.273659 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.273559    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8664c542-7802-4307-a3de-d6a78ccf9436" containerName="maas-api"
Jul 23 18:30:27.273659 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.273645    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8664c542-7802-4307-a3de-d6a78ccf9436" containerName="maas-api"
Jul 23 18:30:27.276788 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.276770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.279497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.279476    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b5680af2-serving-cert\""
Jul 23 18:30:27.279644 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.279585    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:30:27.289447 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.289424    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"]
Jul 23 18:30:27.327518 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.327480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.327705 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.327570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbnh\" (UniqueName: \"kubernetes.io/projected/f93e8624-d395-44a7-8e43-bdafd4558814-kube-api-access-8lbnh\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.327705 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.327620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e8624-d395-44a7-8e43-bdafd4558814-openshift-service-ca\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.428579 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.428548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.428759 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.428602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbnh\" (UniqueName: \"kubernetes.io/projected/f93e8624-d395-44a7-8e43-bdafd4558814-kube-api-access-8lbnh\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.428759 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.428636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e8624-d395-44a7-8e43-bdafd4558814-openshift-service-ca\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.428759 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:30:27.428710    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-serving-cert: secret "maas-api-e2e-derive-b5680af2-serving-cert" not found
Jul 23 18:30:27.428927 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:30:27.428786    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls podName:f93e8624-d395-44a7-8e43-bdafd4558814 nodeName:}" failed. No retries permitted until 2026-07-23 18:30:27.928768114 +0000 UTC m=+1866.718925212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls") pod "maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" (UID: "f93e8624-d395-44a7-8e43-bdafd4558814") : secret "maas-api-e2e-derive-b5680af2-serving-cert" not found
Jul 23 18:30:27.429196 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.429179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e8624-d395-44a7-8e43-bdafd4558814-openshift-service-ca\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.438208 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.438173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbnh\" (UniqueName: \"kubernetes.io/projected/f93e8624-d395-44a7-8e43-bdafd4558814-kube-api-access-8lbnh\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.933573 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.933526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:27.936246 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:27.936219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls\") pod \"maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:28.187148 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:28.187062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:28.333963 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:28.333901    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"]
Jul 23 18:30:28.335664 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:30:28.335634    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93e8624_d395_44a7_8e43_bdafd4558814.slice/crio-8f0fd0024da3591ca9dc54ce27ac3d57f1270aca9cc08a475f6eac72ffe5364f WatchSource:0}: Error finding container 8f0fd0024da3591ca9dc54ce27ac3d57f1270aca9cc08a475f6eac72ffe5364f: Status 404 returned error can't find the container with id 8f0fd0024da3591ca9dc54ce27ac3d57f1270aca9cc08a475f6eac72ffe5364f
Jul 23 18:30:29.048672 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:29.048631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" event={"ID":"f93e8624-d395-44a7-8e43-bdafd4558814","Type":"ContainerStarted","Data":"56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291"}
Jul 23 18:30:29.048859 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:29.048697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" event={"ID":"f93e8624-d395-44a7-8e43-bdafd4558814","Type":"ContainerStarted","Data":"8f0fd0024da3591ca9dc54ce27ac3d57f1270aca9cc08a475f6eac72ffe5364f"}
Jul 23 18:30:29.048859 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:29.048729    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:30:29.066380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:29.066332    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" podStartSLOduration=1.6517458120000001 podStartE2EDuration="2.066316798s" totalImagesPullingTime="414.570986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:30:29.064626199 +0000 UTC m=+1867.854783315" watchObservedRunningTime="2026-07-23 18:30:29.066316798 +0000 UTC m=+1867.856473913"
Jul 23 18:30:35.060738 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:30:35.060707    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:31:02.516498 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.516459    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"]
Jul 23 18:31:02.517125 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.516785    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" podUID="f93e8624-d395-44a7-8e43-bdafd4558814" containerName="maas-api" containerID="cri-o://56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291" gracePeriod=30
Jul 23 18:31:02.773922 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.773866    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:31:02.935765 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.935727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e8624-d395-44a7-8e43-bdafd4558814-openshift-service-ca\") pod \"f93e8624-d395-44a7-8e43-bdafd4558814\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") "
Jul 23 18:31:02.935765 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.935783    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls\") pod \"f93e8624-d395-44a7-8e43-bdafd4558814\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") "
Jul 23 18:31:02.936044 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.935814    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lbnh\" (UniqueName: \"kubernetes.io/projected/f93e8624-d395-44a7-8e43-bdafd4558814-kube-api-access-8lbnh\") pod \"f93e8624-d395-44a7-8e43-bdafd4558814\" (UID: \"f93e8624-d395-44a7-8e43-bdafd4558814\") "
Jul 23 18:31:02.936207 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.936177    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93e8624-d395-44a7-8e43-bdafd4558814-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f93e8624-d395-44a7-8e43-bdafd4558814" (UID: "f93e8624-d395-44a7-8e43-bdafd4558814"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:31:02.938057 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.938029    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f93e8624-d395-44a7-8e43-bdafd4558814" (UID: "f93e8624-d395-44a7-8e43-bdafd4558814"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:31:02.938178 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:02.938099    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93e8624-d395-44a7-8e43-bdafd4558814-kube-api-access-8lbnh" (OuterVolumeSpecName: "kube-api-access-8lbnh") pod "f93e8624-d395-44a7-8e43-bdafd4558814" (UID: "f93e8624-d395-44a7-8e43-bdafd4558814"). InnerVolumeSpecName "kube-api-access-8lbnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:31:03.036545 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.036450    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e8624-d395-44a7-8e43-bdafd4558814-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:31:03.036545 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.036486    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f93e8624-d395-44a7-8e43-bdafd4558814-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:31:03.036545 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.036497    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lbnh\" (UniqueName: \"kubernetes.io/projected/f93e8624-d395-44a7-8e43-bdafd4558814-kube-api-access-8lbnh\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:31:03.235729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.235691    2562 generic.go:356] "Generic (PLEG): container finished" podID="f93e8624-d395-44a7-8e43-bdafd4558814" containerID="56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291" exitCode=0
Jul 23 18:31:03.235927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.235742    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"
Jul 23 18:31:03.235927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.235780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" event={"ID":"f93e8624-d395-44a7-8e43-bdafd4558814","Type":"ContainerDied","Data":"56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291"}
Jul 23 18:31:03.235927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.235844    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg" event={"ID":"f93e8624-d395-44a7-8e43-bdafd4558814","Type":"ContainerDied","Data":"8f0fd0024da3591ca9dc54ce27ac3d57f1270aca9cc08a475f6eac72ffe5364f"}
Jul 23 18:31:03.235927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.235871    2562 scope.go:117] "RemoveContainer" containerID="56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291"
Jul 23 18:31:03.245641 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.245621    2562 scope.go:117] "RemoveContainer" containerID="56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291"
Jul 23 18:31:03.245917 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:31:03.245900    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291\": container with ID starting with 56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291 not found: ID does not exist" containerID="56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291"
Jul 23 18:31:03.246003 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.245927    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291"} err="failed to get container status \"56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291\": rpc error: code = NotFound desc = could not find container \"56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291\": container with ID starting with 56ef308d1301201f8c46adad14626897cac40c339f868bb2791d69c0032ac291 not found: ID does not exist"
Jul 23 18:31:03.257020 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.256984    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"]
Jul 23 18:31:03.260447 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.260425    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5680af2-6ffcc77c-ftjhg"]
Jul 23 18:31:03.608531 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:31:03.608497    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f93e8624-d395-44a7-8e43-bdafd4558814" path="/var/lib/kubelet/pods/f93e8624-d395-44a7-8e43-bdafd4558814/volumes"
Jul 23 18:32:49.935876 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.935844    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"]
Jul 23 18:32:49.936416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.936272    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93e8624-d395-44a7-8e43-bdafd4558814" containerName="maas-api"
Jul 23 18:32:49.936416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.936289    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93e8624-d395-44a7-8e43-bdafd4558814" containerName="maas-api"
Jul 23 18:32:49.936416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.936367    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f93e8624-d395-44a7-8e43-bdafd4558814" containerName="maas-api"
Jul 23 18:32:49.939461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.939444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:49.957752 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.957723    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e318efc2-serving-cert\""
Jul 23 18:32:49.957898 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:49.957878    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:32:50.009016 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.008985    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"]
Jul 23 18:32:50.087800 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.087769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qf2g\" (UniqueName: \"kubernetes.io/projected/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-kube-api-access-7qf2g\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.088023 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.087820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.088023 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.087843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-openshift-service-ca\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.189029 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.188923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.189029 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.188984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-openshift-service-ca\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.189227 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.189049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qf2g\" (UniqueName: \"kubernetes.io/projected/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-kube-api-access-7qf2g\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.189227 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:32:50.189082    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-serving-cert: secret "maas-api-e2e-mt-e318efc2-serving-cert" not found
Jul 23 18:32:50.189227 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:32:50.189153    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls podName:ae4e03b9-f20e-48f8-9711-eef3c49c0e7d nodeName:}" failed. No retries permitted until 2026-07-23 18:32:50.68913628 +0000 UTC m=+2009.479293373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls") pod "maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" (UID: "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d") : secret "maas-api-e2e-mt-e318efc2-serving-cert" not found
Jul 23 18:32:50.189645 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.189626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-openshift-service-ca\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.208284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.208253    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qf2g\" (UniqueName: \"kubernetes.io/projected/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-kube-api-access-7qf2g\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.694291 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.694244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.696870 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.696836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls\") pod \"maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:50.850151 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:50.850114    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:51.008497 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:51.008474    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"]
Jul 23 18:32:51.010486 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:32:51.010450    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4e03b9_f20e_48f8_9711_eef3c49c0e7d.slice/crio-2e8fe9f0a097d87ce1b35b16b31c9bc307c6999fb97197c5c55cbadf6bcfe750 WatchSource:0}: Error finding container 2e8fe9f0a097d87ce1b35b16b31c9bc307c6999fb97197c5c55cbadf6bcfe750: Status 404 returned error can't find the container with id 2e8fe9f0a097d87ce1b35b16b31c9bc307c6999fb97197c5c55cbadf6bcfe750
Jul 23 18:32:51.817966 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:51.817912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" event={"ID":"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d","Type":"ContainerStarted","Data":"dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98"}
Jul 23 18:32:51.818143 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:51.817985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" event={"ID":"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d","Type":"ContainerStarted","Data":"2e8fe9f0a097d87ce1b35b16b31c9bc307c6999fb97197c5c55cbadf6bcfe750"}
Jul 23 18:32:51.818143 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:51.818123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:32:51.849798 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:51.849732    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" podStartSLOduration=2.393667089 podStartE2EDuration="2.849706856s" totalImagesPullingTime="456.039767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:32:51.847803162 +0000 UTC m=+2010.637960288" watchObservedRunningTime="2026-07-23 18:32:51.849706856 +0000 UTC m=+2010.639863972"
Jul 23 18:32:57.829115 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:32:57.829081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:33:11.990693 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:11.990642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"]
Jul 23 18:33:11.991283 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:11.990909    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" podUID="ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" containerName="maas-api" containerID="cri-o://dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98" gracePeriod=30
Jul 23 18:33:12.253237 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.253161    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:33:12.375224 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.375181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qf2g\" (UniqueName: \"kubernetes.io/projected/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-kube-api-access-7qf2g\") pod \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") "
Jul 23 18:33:12.375427 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.375251    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-openshift-service-ca\") pod \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") "
Jul 23 18:33:12.375427 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.375366    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls\") pod \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\" (UID: \"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d\") "
Jul 23 18:33:12.375652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.375622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" (UID: "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:33:12.377652 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.377623    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-kube-api-access-7qf2g" (OuterVolumeSpecName: "kube-api-access-7qf2g") pod "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" (UID: "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d"). InnerVolumeSpecName "kube-api-access-7qf2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:33:12.377763 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.377628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" (UID: "ae4e03b9-f20e-48f8-9711-eef3c49c0e7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:33:12.476910 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.476874    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:33:12.476910 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.476905    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qf2g\" (UniqueName: \"kubernetes.io/projected/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-kube-api-access-7qf2g\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:33:12.476910 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.476914    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:33:12.936074 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.936036    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" containerID="dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98" exitCode=0
Jul 23 18:33:12.936298 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.936095    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"
Jul 23 18:33:12.936298 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.936140    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" event={"ID":"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d","Type":"ContainerDied","Data":"dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98"}
Jul 23 18:33:12.936298 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.936205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j" event={"ID":"ae4e03b9-f20e-48f8-9711-eef3c49c0e7d","Type":"ContainerDied","Data":"2e8fe9f0a097d87ce1b35b16b31c9bc307c6999fb97197c5c55cbadf6bcfe750"}
Jul 23 18:33:12.936298 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.936235    2562 scope.go:117] "RemoveContainer" containerID="dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98"
Jul 23 18:33:12.945858 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.945835    2562 scope.go:117] "RemoveContainer" containerID="dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98"
Jul 23 18:33:12.946199 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:33:12.946170    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98\": container with ID starting with dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98 not found: ID does not exist" containerID="dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98"
Jul 23 18:33:12.946265 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.946205    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98"} err="failed to get container status \"dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98\": rpc error: code = NotFound desc = could not find container \"dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98\": container with ID starting with dfd5b8b721a51783ef95155667ae785dbb1502590d583ce4a8eb3193d184fe98 not found: ID does not exist"
Jul 23 18:33:12.957496 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.957464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"]
Jul 23 18:33:12.960667 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:12.960638    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e318efc2-546567dbfb-8lr9j"]
Jul 23 18:33:13.609174 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:13.609143    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" path="/var/lib/kubelet/pods/ae4e03b9-f20e-48f8-9711-eef3c49c0e7d/volumes"
Jul 23 18:33:47.833905 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.833769    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"]
Jul 23 18:33:47.834816 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.834571    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" containerName="maas-api"
Jul 23 18:33:47.834816 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.834595    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" containerName="maas-api"
Jul 23 18:33:47.834816 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.834702    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae4e03b9-f20e-48f8-9711-eef3c49c0e7d" containerName="maas-api"
Jul 23 18:33:47.838146 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.838122    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.840306 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.840283    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08efc605-serving-cert\""
Jul 23 18:33:47.840431 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.840362    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:33:47.848270 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.848248    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"]
Jul 23 18:33:47.893698 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.893660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-openshift-service-ca\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.893886 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.893758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8ng\" (UniqueName: \"kubernetes.io/projected/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-kube-api-access-wf8ng\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.893886 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.893800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.994755 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.994708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8ng\" (UniqueName: \"kubernetes.io/projected/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-kube-api-access-wf8ng\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.995001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.994817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.995001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.994921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-openshift-service-ca\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:47.995150 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:33:47.995006    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-serving-cert: secret "maas-api-e2e-mt-08efc605-serving-cert" not found
Jul 23 18:33:47.995150 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:33:47.995100    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls podName:d55f3c37-4a80-4fbc-b438-35121f7c5fe7 nodeName:}" failed. No retries permitted until 2026-07-23 18:33:48.495076469 +0000 UTC m=+2067.285233577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls") pod "maas-api-e2e-mt-08efc605-ccd46fb94-dx655" (UID: "d55f3c37-4a80-4fbc-b438-35121f7c5fe7") : secret "maas-api-e2e-mt-08efc605-serving-cert" not found
Jul 23 18:33:47.995547 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:47.995527    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-openshift-service-ca\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:48.003854 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:48.003825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8ng\" (UniqueName: \"kubernetes.io/projected/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-kube-api-access-wf8ng\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:48.500300 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:48.500266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:48.502934 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:48.502908    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls\") pod \"maas-api-e2e-mt-08efc605-ccd46fb94-dx655\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:48.749370 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:48.749332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:48.901988 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:48.895860    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"]
Jul 23 18:33:48.903448 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:33:48.903410    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55f3c37_4a80_4fbc_b438_35121f7c5fe7.slice/crio-3a74f249d4931ddc534afbcd92018f7109e6e16ec6501e5344cc4c695c76ff51 WatchSource:0}: Error finding container 3a74f249d4931ddc534afbcd92018f7109e6e16ec6501e5344cc4c695c76ff51: Status 404 returned error can't find the container with id 3a74f249d4931ddc534afbcd92018f7109e6e16ec6501e5344cc4c695c76ff51
Jul 23 18:33:48.905402 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:48.905378    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:33:49.135219 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:49.135181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655" event={"ID":"d55f3c37-4a80-4fbc-b438-35121f7c5fe7","Type":"ContainerStarted","Data":"3a74f249d4931ddc534afbcd92018f7109e6e16ec6501e5344cc4c695c76ff51"}
Jul 23 18:33:50.142347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:50.142313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655" event={"ID":"d55f3c37-4a80-4fbc-b438-35121f7c5fe7","Type":"ContainerStarted","Data":"bc678e8a7f40379c2c856db0ec145f02f9387eb7cad3ea9690c6d96c103bc89f"}
Jul 23 18:33:50.142803 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:50.142442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:33:50.160867 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:50.160811    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655" podStartSLOduration=2.720166089 podStartE2EDuration="3.16079527s" totalImagesPullingTime="440.629181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:33:50.159735277 +0000 UTC m=+2068.949892394" watchObservedRunningTime="2026-07-23 18:33:50.16079527 +0000 UTC m=+2068.950952384"
Jul 23 18:33:56.154600 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:33:56.154568    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:34:21.753019 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:21.752991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:34:21.763986 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:21.763931    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:34:36.223583 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.223550    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"]
Jul 23 18:34:36.224213 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.223781    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655" podUID="d55f3c37-4a80-4fbc-b438-35121f7c5fe7" containerName="maas-api" containerID="cri-o://bc678e8a7f40379c2c856db0ec145f02f9387eb7cad3ea9690c6d96c103bc89f" gracePeriod=30
Jul 23 18:34:36.394353 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.394313    2562 generic.go:356] "Generic (PLEG): container finished" podID="d55f3c37-4a80-4fbc-b438-35121f7c5fe7" containerID="bc678e8a7f40379c2c856db0ec145f02f9387eb7cad3ea9690c6d96c103bc89f" exitCode=0
Jul 23 18:34:36.394567 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.394433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655" event={"ID":"d55f3c37-4a80-4fbc-b438-35121f7c5fe7","Type":"ContainerDied","Data":"bc678e8a7f40379c2c856db0ec145f02f9387eb7cad3ea9690c6d96c103bc89f"}
Jul 23 18:34:36.481371 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.481312    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:34:36.614978 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.614913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-openshift-service-ca\") pod \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") "
Jul 23 18:34:36.614978 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.614971    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls\") pod \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") "
Jul 23 18:34:36.615232 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.615020    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf8ng\" (UniqueName: \"kubernetes.io/projected/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-kube-api-access-wf8ng\") pod \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\" (UID: \"d55f3c37-4a80-4fbc-b438-35121f7c5fe7\") "
Jul 23 18:34:36.615296 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.615278    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d55f3c37-4a80-4fbc-b438-35121f7c5fe7" (UID: "d55f3c37-4a80-4fbc-b438-35121f7c5fe7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:34:36.617218 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.617193    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-kube-api-access-wf8ng" (OuterVolumeSpecName: "kube-api-access-wf8ng") pod "d55f3c37-4a80-4fbc-b438-35121f7c5fe7" (UID: "d55f3c37-4a80-4fbc-b438-35121f7c5fe7"). InnerVolumeSpecName "kube-api-access-wf8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:34:36.617321 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.617236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d55f3c37-4a80-4fbc-b438-35121f7c5fe7" (UID: "d55f3c37-4a80-4fbc-b438-35121f7c5fe7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:34:36.715933 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.715892    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:34:36.715933 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.715925    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:34:36.715933 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:36.715935    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8ng\" (UniqueName: \"kubernetes.io/projected/d55f3c37-4a80-4fbc-b438-35121f7c5fe7-kube-api-access-wf8ng\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:34:37.402591 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:37.402543    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"
Jul 23 18:34:37.403058 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:37.402538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655" event={"ID":"d55f3c37-4a80-4fbc-b438-35121f7c5fe7","Type":"ContainerDied","Data":"3a74f249d4931ddc534afbcd92018f7109e6e16ec6501e5344cc4c695c76ff51"}
Jul 23 18:34:37.403058 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:37.402695    2562 scope.go:117] "RemoveContainer" containerID="bc678e8a7f40379c2c856db0ec145f02f9387eb7cad3ea9690c6d96c103bc89f"
Jul 23 18:34:37.427597 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:37.427571    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"]
Jul 23 18:34:37.431902 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:37.431876    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08efc605-ccd46fb94-dx655"]
Jul 23 18:34:37.609027 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:34:37.608984    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55f3c37-4a80-4fbc-b438-35121f7c5fe7" path="/var/lib/kubelet/pods/d55f3c37-4a80-4fbc-b438-35121f7c5fe7/volumes"
Jul 23 18:35:21.829959 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:35:21.829876    2562 scope.go:117] "RemoveContainer" containerID="0b0a381c11bcda1d14e3dec1104b1773e8d05b226ab436033f91be3e256c2531"
Jul 23 18:36:31.705560 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.705523    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"]
Jul 23 18:36:31.706163 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.706092    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55f3c37-4a80-4fbc-b438-35121f7c5fe7" containerName="maas-api"
Jul 23 18:36:31.706163 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.706112    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55f3c37-4a80-4fbc-b438-35121f7c5fe7" containerName="maas-api"
Jul 23 18:36:31.706287 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.706229    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55f3c37-4a80-4fbc-b438-35121f7c5fe7" containerName="maas-api"
Jul 23 18:36:31.709369 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.709348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.711973 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.711932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-816eca-serving-cert\""
Jul 23 18:36:31.712091 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.712012    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:36:31.720401 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.720373    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"]
Jul 23 18:36:31.823309 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.823266    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.823309 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.823316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqnq\" (UniqueName: \"kubernetes.io/projected/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-kube-api-access-jpqnq\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.823572 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.823466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.924202 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.924166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.924428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.924223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.924428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.924340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqnq\" (UniqueName: \"kubernetes.io/projected/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-kube-api-access-jpqnq\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.924428 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:36:31.924346    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-serving-cert: secret "maas-api-e2e-inf-a-816eca-serving-cert" not found
Jul 23 18:36:31.924534 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:36:31.924473    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls podName:ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a nodeName:}" failed. No retries permitted until 2026-07-23 18:36:32.424452505 +0000 UTC m=+2231.214609615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls") pod "maas-api-e2e-inf-a-816eca-5778c98755-svx48" (UID: "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a") : secret "maas-api-e2e-inf-a-816eca-serving-cert" not found
Jul 23 18:36:31.924832 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.924815    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:31.933445 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:31.933418    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqnq\" (UniqueName: \"kubernetes.io/projected/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-kube-api-access-jpqnq\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:32.429152 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:32.429114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:32.429339 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:36:32.429277    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-serving-cert: secret "maas-api-e2e-inf-a-816eca-serving-cert" not found
Jul 23 18:36:32.429383 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:36:32.429355    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls podName:ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a nodeName:}" failed. No retries permitted until 2026-07-23 18:36:33.429339733 +0000 UTC m=+2232.219496828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls") pod "maas-api-e2e-inf-a-816eca-5778c98755-svx48" (UID: "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a") : secret "maas-api-e2e-inf-a-816eca-serving-cert" not found
Jul 23 18:36:33.439139 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:33.439107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:33.441708 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:33.441685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") pod \"maas-api-e2e-inf-a-816eca-5778c98755-svx48\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:33.521752 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:33.521714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:33.663936 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:33.663910    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"]
Jul 23 18:36:33.665535 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:36:33.665507    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd0bd04_b1eb_4ff9_a7e5_1d42504b042a.slice/crio-0c726895e61fb29e1d5e57a20408eb0d585462491da62c9a814f3e7b5e5da96a WatchSource:0}: Error finding container 0c726895e61fb29e1d5e57a20408eb0d585462491da62c9a814f3e7b5e5da96a: Status 404 returned error can't find the container with id 0c726895e61fb29e1d5e57a20408eb0d585462491da62c9a814f3e7b5e5da96a
Jul 23 18:36:34.023284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:34.023250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48" event={"ID":"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a","Type":"ContainerStarted","Data":"0c726895e61fb29e1d5e57a20408eb0d585462491da62c9a814f3e7b5e5da96a"}
Jul 23 18:36:35.031284 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:35.031249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48" event={"ID":"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a","Type":"ContainerStarted","Data":"9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff"}
Jul 23 18:36:35.031681 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:35.031346    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:35.053199 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:35.053150    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48" podStartSLOduration=3.624692153 podStartE2EDuration="4.053135486s" totalImagesPullingTime="428.443333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:36:35.05123046 +0000 UTC m=+2233.841387576" watchObservedRunningTime="2026-07-23 18:36:35.053135486 +0000 UTC m=+2233.843292600"
Jul 23 18:36:39.883114 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:39.883079    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"]
Jul 23 18:36:39.886910 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:39.886887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:39.889482 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:39.889454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-629a5d-serving-cert\""
Jul 23 18:36:39.902994 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:39.902917    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"]
Jul 23 18:36:40.003805 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.003766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-maas-api-tls\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.003805 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.003806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvfp\" (UniqueName: \"kubernetes.io/projected/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-kube-api-access-dmvfp\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.004107 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.003850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-openshift-service-ca\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.105275 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.105234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-maas-api-tls\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.105275 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.105269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvfp\" (UniqueName: \"kubernetes.io/projected/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-kube-api-access-dmvfp\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.105483 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.105332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-openshift-service-ca\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.106071 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.106039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-openshift-service-ca\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.108031 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.108009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-maas-api-tls\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.114269 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.114247    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvfp\" (UniqueName: \"kubernetes.io/projected/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-kube-api-access-dmvfp\") pod \"maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.201060 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.200972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:40.363074 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:40.360542    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"]
Jul 23 18:36:40.363447 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:36:40.363415    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ece4d1_7b72_4ffa_9da7_138cd4cdea64.slice/crio-a10b6beacf1a6f75b8da0b785fdddaf61bf0fd2c727ab5cc1ce95b82de5492d5 WatchSource:0}: Error finding container a10b6beacf1a6f75b8da0b785fdddaf61bf0fd2c727ab5cc1ce95b82de5492d5: Status 404 returned error can't find the container with id a10b6beacf1a6f75b8da0b785fdddaf61bf0fd2c727ab5cc1ce95b82de5492d5
Jul 23 18:36:41.042762 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:41.042681    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:36:41.065162 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:41.065125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r" event={"ID":"19ece4d1-7b72-4ffa-9da7-138cd4cdea64","Type":"ContainerStarted","Data":"fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5"}
Jul 23 18:36:41.065356 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:41.065174    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r" event={"ID":"19ece4d1-7b72-4ffa-9da7-138cd4cdea64","Type":"ContainerStarted","Data":"a10b6beacf1a6f75b8da0b785fdddaf61bf0fd2c727ab5cc1ce95b82de5492d5"}
Jul 23 18:36:41.065356 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:41.065255    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:36:41.085934 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:41.085866    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r" podStartSLOduration=1.6666734600000002 podStartE2EDuration="2.085844718s" totalImagesPullingTime="419.171258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:36:41.08222066 +0000 UTC m=+2239.872377792" watchObservedRunningTime="2026-07-23 18:36:41.085844718 +0000 UTC m=+2239.876001834"
Jul 23 18:36:47.076152 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:36:47.076071    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:38:20.658349 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.658308    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"]
Jul 23 18:38:20.661830 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.661809    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:20.664181 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.664157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:38:20.669148 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.669124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"]
Jul 23 18:38:20.769565 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.769526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955b8bf9-605c-497c-bf6e-ad851da2d4a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc\" (UID: \"955b8bf9-605c-497c-bf6e-ad851da2d4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:20.870439 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.870395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955b8bf9-605c-497c-bf6e-ad851da2d4a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc\" (UID: \"955b8bf9-605c-497c-bf6e-ad851da2d4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:20.871050 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.871032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955b8bf9-605c-497c-bf6e-ad851da2d4a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc\" (UID: \"955b8bf9-605c-497c-bf6e-ad851da2d4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:20.975494 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:20.975411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:21.115727 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:21.115692    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"]
Jul 23 18:38:21.117668 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:38:21.117643    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955b8bf9_605c_497c_bf6e_ad851da2d4a0.slice/crio-67d006a85be8b4da49df37030ed4631d6ead0d0b6326c18d7a5137c1b7fb9355 WatchSource:0}: Error finding container 67d006a85be8b4da49df37030ed4631d6ead0d0b6326c18d7a5137c1b7fb9355: Status 404 returned error can't find the container with id 67d006a85be8b4da49df37030ed4631d6ead0d0b6326c18d7a5137c1b7fb9355
Jul 23 18:38:21.622701 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:21.622607    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc" event={"ID":"955b8bf9-605c-497c-bf6e-ad851da2d4a0","Type":"ContainerStarted","Data":"df2f5e1205f6e4770361d2bb443de0509ea5ab17cb794d1566fb7e8ba3f53a01"}
Jul 23 18:38:21.622701 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:21.622657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc" event={"ID":"955b8bf9-605c-497c-bf6e-ad851da2d4a0","Type":"ContainerStarted","Data":"67d006a85be8b4da49df37030ed4631d6ead0d0b6326c18d7a5137c1b7fb9355"}
Jul 23 18:38:21.637124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:21.637071    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc" podStartSLOduration=1.637052612 podStartE2EDuration="1.637052612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:38:21.636225192 +0000 UTC m=+2340.426382308" watchObservedRunningTime="2026-07-23 18:38:21.637052612 +0000 UTC m=+2340.427209727"
Jul 23 18:38:22.629429 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:22.629397    2562 generic.go:356] "Generic (PLEG): container finished" podID="955b8bf9-605c-497c-bf6e-ad851da2d4a0" containerID="df2f5e1205f6e4770361d2bb443de0509ea5ab17cb794d1566fb7e8ba3f53a01" exitCode=0
Jul 23 18:38:22.631759 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:22.629463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc" event={"ID":"955b8bf9-605c-497c-bf6e-ad851da2d4a0","Type":"ContainerDied","Data":"df2f5e1205f6e4770361d2bb443de0509ea5ab17cb794d1566fb7e8ba3f53a01"}
Jul 23 18:38:23.772384 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:23.772356    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:23.897799 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:23.897709    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955b8bf9-605c-497c-bf6e-ad851da2d4a0-maas-api-service-ca\") pod \"955b8bf9-605c-497c-bf6e-ad851da2d4a0\" (UID: \"955b8bf9-605c-497c-bf6e-ad851da2d4a0\") "
Jul 23 18:38:23.898092 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:23.898069    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955b8bf9-605c-497c-bf6e-ad851da2d4a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "955b8bf9-605c-497c-bf6e-ad851da2d4a0" (UID: "955b8bf9-605c-497c-bf6e-ad851da2d4a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:38:23.999059 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:23.999026    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955b8bf9-605c-497c-bf6e-ad851da2d4a0-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:24.641481 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:24.641452    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"
Jul 23 18:38:24.641481 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:24.641465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc" event={"ID":"955b8bf9-605c-497c-bf6e-ad851da2d4a0","Type":"ContainerDied","Data":"67d006a85be8b4da49df37030ed4631d6ead0d0b6326c18d7a5137c1b7fb9355"}
Jul 23 18:38:24.641738 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:24.641519    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d006a85be8b4da49df37030ed4631d6ead0d0b6326c18d7a5137c1b7fb9355"
Jul 23 18:38:31.228629 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.228595    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"]
Jul 23 18:38:31.229824 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.228958    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48" podUID="ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" containerName="maas-api" containerID="cri-o://9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff" gracePeriod=30
Jul 23 18:38:31.480638 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.480581    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:38:31.563272 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.563237    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-openshift-service-ca\") pod \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") "
Jul 23 18:38:31.563482 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.563308    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") pod \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") "
Jul 23 18:38:31.563482 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.563377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpqnq\" (UniqueName: \"kubernetes.io/projected/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-kube-api-access-jpqnq\") pod \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\" (UID: \"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a\") "
Jul 23 18:38:31.563657 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.563626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" (UID: "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:38:31.565617 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.565582    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" (UID: "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:38:31.565617 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.565598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-kube-api-access-jpqnq" (OuterVolumeSpecName: "kube-api-access-jpqnq") pod "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" (UID: "ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a"). InnerVolumeSpecName "kube-api-access-jpqnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:38:31.664380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.664341    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:31.664380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.664375    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqnq\" (UniqueName: \"kubernetes.io/projected/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-kube-api-access-jpqnq\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:31.664380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.664385    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:31.680286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.680247    2562 generic.go:356] "Generic (PLEG): container finished" podID="ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" containerID="9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff" exitCode=0
Jul 23 18:38:31.680419 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.680299    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"
Jul 23 18:38:31.680419 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.680306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48" event={"ID":"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a","Type":"ContainerDied","Data":"9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff"}
Jul 23 18:38:31.680419 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.680359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48" event={"ID":"ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a","Type":"ContainerDied","Data":"0c726895e61fb29e1d5e57a20408eb0d585462491da62c9a814f3e7b5e5da96a"}
Jul 23 18:38:31.680419 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.680387    2562 scope.go:117] "RemoveContainer" containerID="9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff"
Jul 23 18:38:31.691355 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.691331    2562 scope.go:117] "RemoveContainer" containerID="9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff"
Jul 23 18:38:31.691629 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:38:31.691608    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff\": container with ID starting with 9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff not found: ID does not exist" containerID="9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff"
Jul 23 18:38:31.691673 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.691638    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff"} err="failed to get container status \"9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff\": rpc error: code = NotFound desc = could not find container \"9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff\": container with ID starting with 9021780d92d1dc69cfbd1d71fed1187548deb16a1e5398b6adaaeba3d1ea73ff not found: ID does not exist"
Jul 23 18:38:31.700421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.700359    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"]
Jul 23 18:38:31.703954 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:31.703914    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-816eca-5778c98755-svx48"]
Jul 23 18:38:33.609104 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:33.609065    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" path="/var/lib/kubelet/pods/ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a/volumes"
Jul 23 18:38:35.764999 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:35.764967    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"]
Jul 23 18:38:35.767379 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:35.767355    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-816eca-nwcrc"]
Jul 23 18:38:37.609447 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:37.609415    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955b8bf9-605c-497c-bf6e-ad851da2d4a0" path="/var/lib/kubelet/pods/955b8bf9-605c-497c-bf6e-ad851da2d4a0/volumes"
Jul 23 18:38:42.744068 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744029    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"]
Jul 23 18:38:42.744443 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744423    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955b8bf9-605c-497c-bf6e-ad851da2d4a0" containerName="revoke-keys"
Jul 23 18:38:42.744443 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744433    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="955b8bf9-605c-497c-bf6e-ad851da2d4a0" containerName="revoke-keys"
Jul 23 18:38:42.744512 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744453    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" containerName="maas-api"
Jul 23 18:38:42.744512 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744459    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" containerName="maas-api"
Jul 23 18:38:42.744583 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744517    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd0bd04-b1eb-4ff9-a7e5-1d42504b042a" containerName="maas-api"
Jul 23 18:38:42.744583 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.744526    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="955b8bf9-605c-497c-bf6e-ad851da2d4a0" containerName="revoke-keys"
Jul 23 18:38:42.748990 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.748973    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:42.751630 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.751607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:38:42.754662 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.754633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"]
Jul 23 18:38:42.869514 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.869478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f733cecd-2896-4228-880c-c575cbcd91fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x\" (UID: \"f733cecd-2896-4228-880c-c575cbcd91fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:42.970656 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.970624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f733cecd-2896-4228-880c-c575cbcd91fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x\" (UID: \"f733cecd-2896-4228-880c-c575cbcd91fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:42.971241 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:42.971224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f733cecd-2896-4228-880c-c575cbcd91fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x\" (UID: \"f733cecd-2896-4228-880c-c575cbcd91fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:43.061001 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:43.060963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:43.194005 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:43.193974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"]
Jul 23 18:38:43.196789 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:38:43.196754    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf733cecd_2896_4228_880c_c575cbcd91fc.slice/crio-2d94fe560f9b3a3add0270028fa561138c52c88b8e17ac698cad54d91771aeaf WatchSource:0}: Error finding container 2d94fe560f9b3a3add0270028fa561138c52c88b8e17ac698cad54d91771aeaf: Status 404 returned error can't find the container with id 2d94fe560f9b3a3add0270028fa561138c52c88b8e17ac698cad54d91771aeaf
Jul 23 18:38:43.745906 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:43.745799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x" event={"ID":"f733cecd-2896-4228-880c-c575cbcd91fc","Type":"ContainerStarted","Data":"29ed3d30503b2d49d7f67d7044db84976422e9e2f08a77b4379392d7aea59261"}
Jul 23 18:38:43.745906 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:43.745875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x" event={"ID":"f733cecd-2896-4228-880c-c575cbcd91fc","Type":"ContainerStarted","Data":"2d94fe560f9b3a3add0270028fa561138c52c88b8e17ac698cad54d91771aeaf"}
Jul 23 18:38:43.762829 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:43.762779    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x" podStartSLOduration=1.762761399 podStartE2EDuration="1.762761399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:38:43.761903802 +0000 UTC m=+2362.552060934" watchObservedRunningTime="2026-07-23 18:38:43.762761399 +0000 UTC m=+2362.552918704"
Jul 23 18:38:44.752641 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:44.752605    2562 generic.go:356] "Generic (PLEG): container finished" podID="f733cecd-2896-4228-880c-c575cbcd91fc" containerID="29ed3d30503b2d49d7f67d7044db84976422e9e2f08a77b4379392d7aea59261" exitCode=0
Jul 23 18:38:44.753045 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:44.752695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x" event={"ID":"f733cecd-2896-4228-880c-c575cbcd91fc","Type":"ContainerDied","Data":"29ed3d30503b2d49d7f67d7044db84976422e9e2f08a77b4379392d7aea59261"}
Jul 23 18:38:45.898841 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:45.898814    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:45.996093 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:45.996061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f733cecd-2896-4228-880c-c575cbcd91fc-maas-api-service-ca\") pod \"f733cecd-2896-4228-880c-c575cbcd91fc\" (UID: \"f733cecd-2896-4228-880c-c575cbcd91fc\") "
Jul 23 18:38:45.996451 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:45.996425    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f733cecd-2896-4228-880c-c575cbcd91fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f733cecd-2896-4228-880c-c575cbcd91fc" (UID: "f733cecd-2896-4228-880c-c575cbcd91fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:38:46.096920 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:46.096822    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f733cecd-2896-4228-880c-c575cbcd91fc-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:46.765912 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:46.765881    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x" event={"ID":"f733cecd-2896-4228-880c-c575cbcd91fc","Type":"ContainerDied","Data":"2d94fe560f9b3a3add0270028fa561138c52c88b8e17ac698cad54d91771aeaf"}
Jul 23 18:38:46.765912 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:46.765904    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"
Jul 23 18:38:46.766150 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:46.765926    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d94fe560f9b3a3add0270028fa561138c52c88b8e17ac698cad54d91771aeaf"
Jul 23 18:38:53.154530 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.154496    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"]
Jul 23 18:38:53.154994 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.154744    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r" podUID="19ece4d1-7b72-4ffa-9da7-138cd4cdea64" containerName="maas-api" containerID="cri-o://fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5" gracePeriod=30
Jul 23 18:38:53.407160 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.407093    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:38:53.460989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.460932    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-openshift-service-ca\") pod \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") "
Jul 23 18:38:53.460989 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.460983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmvfp\" (UniqueName: \"kubernetes.io/projected/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-kube-api-access-dmvfp\") pod \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") "
Jul 23 18:38:53.461200 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.461001    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-maas-api-tls\") pod \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\" (UID: \"19ece4d1-7b72-4ffa-9da7-138cd4cdea64\") "
Jul 23 18:38:53.461362 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.461333    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19ece4d1-7b72-4ffa-9da7-138cd4cdea64" (UID: "19ece4d1-7b72-4ffa-9da7-138cd4cdea64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:38:53.463240 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.463206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19ece4d1-7b72-4ffa-9da7-138cd4cdea64" (UID: "19ece4d1-7b72-4ffa-9da7-138cd4cdea64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:38:53.463357 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.463269    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-kube-api-access-dmvfp" (OuterVolumeSpecName: "kube-api-access-dmvfp") pod "19ece4d1-7b72-4ffa-9da7-138cd4cdea64" (UID: "19ece4d1-7b72-4ffa-9da7-138cd4cdea64"). InnerVolumeSpecName "kube-api-access-dmvfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:38:53.561762 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.561726    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:53.561762 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.561754    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmvfp\" (UniqueName: \"kubernetes.io/projected/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-kube-api-access-dmvfp\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:53.561762 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.561764    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ece4d1-7b72-4ffa-9da7-138cd4cdea64-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:38:53.805426 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.805386    2562 generic.go:356] "Generic (PLEG): container finished" podID="19ece4d1-7b72-4ffa-9da7-138cd4cdea64" containerID="fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5" exitCode=0
Jul 23 18:38:53.805598 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.805434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r" event={"ID":"19ece4d1-7b72-4ffa-9da7-138cd4cdea64","Type":"ContainerDied","Data":"fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5"}
Jul 23 18:38:53.805598 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.805473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r" event={"ID":"19ece4d1-7b72-4ffa-9da7-138cd4cdea64","Type":"ContainerDied","Data":"a10b6beacf1a6f75b8da0b785fdddaf61bf0fd2c727ab5cc1ce95b82de5492d5"}
Jul 23 18:38:53.805598 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.805493    2562 scope.go:117] "RemoveContainer" containerID="fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5"
Jul 23 18:38:53.805598 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.805436    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"
Jul 23 18:38:53.816124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.816106    2562 scope.go:117] "RemoveContainer" containerID="fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5"
Jul 23 18:38:53.816455 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:38:53.816431    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5\": container with ID starting with fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5 not found: ID does not exist" containerID="fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5"
Jul 23 18:38:53.816516 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.816468    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5"} err="failed to get container status \"fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5\": rpc error: code = NotFound desc = could not find container \"fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5\": container with ID starting with fc853cdf3f755b8d4a4c514dc04d0c7a0da925fa42d7dd0c5e67c5e1a56498d5 not found: ID does not exist"
Jul 23 18:38:53.823920 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.823883    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"]
Jul 23 18:38:53.826043 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:53.826020    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629a5d-5646dcd889-9qq7r"]
Jul 23 18:38:55.609267 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:55.609224    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19ece4d1-7b72-4ffa-9da7-138cd4cdea64" path="/var/lib/kubelet/pods/19ece4d1-7b72-4ffa-9da7-138cd4cdea64/volumes"
Jul 23 18:38:57.811529 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:57.811498    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"]
Jul 23 18:38:57.814061 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:57.814036    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629a5d-zrj7x"]
Jul 23 18:38:59.613799 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:38:59.608855    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f733cecd-2896-4228-880c-c575cbcd91fc" path="/var/lib/kubelet/pods/f733cecd-2896-4228-880c-c575cbcd91fc/volumes"
Jul 23 18:39:08.516167 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516132    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"]
Jul 23 18:39:08.516761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516530    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f733cecd-2896-4228-880c-c575cbcd91fc" containerName="revoke-keys"
Jul 23 18:39:08.516761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516541    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f733cecd-2896-4228-880c-c575cbcd91fc" containerName="revoke-keys"
Jul 23 18:39:08.516761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516556    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ece4d1-7b72-4ffa-9da7-138cd4cdea64" containerName="maas-api"
Jul 23 18:39:08.516761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516560    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ece4d1-7b72-4ffa-9da7-138cd4cdea64" containerName="maas-api"
Jul 23 18:39:08.516761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516617    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ece4d1-7b72-4ffa-9da7-138cd4cdea64" containerName="maas-api"
Jul 23 18:39:08.516761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.516625    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f733cecd-2896-4228-880c-c575cbcd91fc" containerName="revoke-keys"
Jul 23 18:39:08.520191 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.520170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.522851 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.522832    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-22682a-serving-cert\""
Jul 23 18:39:08.522851 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.522844    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:39:08.530921 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.530895    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"]
Jul 23 18:39:08.594816 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.594773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c521241-546a-4a35-a9f1-a712a0a2bbef-openshift-service-ca\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.594816 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.594820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5m4h\" (UniqueName: \"kubernetes.io/projected/8c521241-546a-4a35-a9f1-a712a0a2bbef-kube-api-access-c5m4h\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.595081 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.594925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c521241-546a-4a35-a9f1-a712a0a2bbef-maas-api-tls\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.696367 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.696330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c521241-546a-4a35-a9f1-a712a0a2bbef-openshift-service-ca\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.696367 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.696371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5m4h\" (UniqueName: \"kubernetes.io/projected/8c521241-546a-4a35-a9f1-a712a0a2bbef-kube-api-access-c5m4h\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.696628 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.696442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c521241-546a-4a35-a9f1-a712a0a2bbef-maas-api-tls\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.697034 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.697009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c521241-546a-4a35-a9f1-a712a0a2bbef-openshift-service-ca\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.699088 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.699066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c521241-546a-4a35-a9f1-a712a0a2bbef-maas-api-tls\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.706221 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.706195    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5m4h\" (UniqueName: \"kubernetes.io/projected/8c521241-546a-4a35-a9f1-a712a0a2bbef-kube-api-access-c5m4h\") pod \"maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.833847 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.833742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:08.980119 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.980093    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"]
Jul 23 18:39:08.982130 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:39:08.982101    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c521241_546a_4a35_a9f1_a712a0a2bbef.slice/crio-9fc4611310edda22867f0acde48f0c4d6f7ebc90855edbb669eee168f512924d WatchSource:0}: Error finding container 9fc4611310edda22867f0acde48f0c4d6f7ebc90855edbb669eee168f512924d: Status 404 returned error can't find the container with id 9fc4611310edda22867f0acde48f0c4d6f7ebc90855edbb669eee168f512924d
Jul 23 18:39:08.983424 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:08.983407    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:39:09.896538 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:09.896434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m" event={"ID":"8c521241-546a-4a35-a9f1-a712a0a2bbef","Type":"ContainerStarted","Data":"dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f"}
Jul 23 18:39:09.896538 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:09.896500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m" event={"ID":"8c521241-546a-4a35-a9f1-a712a0a2bbef","Type":"ContainerStarted","Data":"9fc4611310edda22867f0acde48f0c4d6f7ebc90855edbb669eee168f512924d"}
Jul 23 18:39:09.897063 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:09.896605    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:09.921702 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:09.921648    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m" podStartSLOduration=1.342976799 podStartE2EDuration="1.921630625s" totalImagesPullingTime="578.653826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:09.92129258 +0000 UTC m=+2388.711449693" watchObservedRunningTime="2026-07-23 18:39:09.921630625 +0000 UTC m=+2388.711787739"
Jul 23 18:39:15.907839 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:15.907807    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:16.490764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.490729    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"]
Jul 23 18:39:16.494706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.494687    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.496915 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.496889    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f77b55-serving-cert\""
Jul 23 18:39:16.503766 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.503739    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"]
Jul 23 18:39:16.569491 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.569447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290b274d-c871-452d-b90f-304c54f09ad4-maas-api-tls\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.569696 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.569558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7fs5\" (UniqueName: \"kubernetes.io/projected/290b274d-c871-452d-b90f-304c54f09ad4-kube-api-access-f7fs5\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.569696 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.569624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b274d-c871-452d-b90f-304c54f09ad4-openshift-service-ca\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.670764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.670726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b274d-c871-452d-b90f-304c54f09ad4-openshift-service-ca\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.670995 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.670776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290b274d-c871-452d-b90f-304c54f09ad4-maas-api-tls\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.670995 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.670830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7fs5\" (UniqueName: \"kubernetes.io/projected/290b274d-c871-452d-b90f-304c54f09ad4-kube-api-access-f7fs5\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.671476 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.671452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b274d-c871-452d-b90f-304c54f09ad4-openshift-service-ca\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.673361 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.673339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290b274d-c871-452d-b90f-304c54f09ad4-maas-api-tls\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.678975 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.678933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7fs5\" (UniqueName: \"kubernetes.io/projected/290b274d-c871-452d-b90f-304c54f09ad4-kube-api-access-f7fs5\") pod \"maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.809201 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.809156    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:16.954791 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:16.954766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"]
Jul 23 18:39:16.956758 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:39:16.956733    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290b274d_c871_452d_b90f_304c54f09ad4.slice/crio-8bb7698b3130a842f204333f9a6689cafefe39678fed6d191b0c7f08baa542d0 WatchSource:0}: Error finding container 8bb7698b3130a842f204333f9a6689cafefe39678fed6d191b0c7f08baa542d0: Status 404 returned error can't find the container with id 8bb7698b3130a842f204333f9a6689cafefe39678fed6d191b0c7f08baa542d0
Jul 23 18:39:17.946280 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:17.946240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5" event={"ID":"290b274d-c871-452d-b90f-304c54f09ad4","Type":"ContainerStarted","Data":"d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0"}
Jul 23 18:39:17.946503 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:17.946306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5" event={"ID":"290b274d-c871-452d-b90f-304c54f09ad4","Type":"ContainerStarted","Data":"8bb7698b3130a842f204333f9a6689cafefe39678fed6d191b0c7f08baa542d0"}
Jul 23 18:39:17.946503 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:17.946379    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:17.964072 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:17.964024    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5" podStartSLOduration=1.627421163 podStartE2EDuration="1.964009141s" totalImagesPullingTime="336.587978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:17.962080438 +0000 UTC m=+2396.752237553" watchObservedRunningTime="2026-07-23 18:39:17.964009141 +0000 UTC m=+2396.754166255"
Jul 23 18:39:21.788881 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:21.788851    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:39:21.801510 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:21.801484    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:39:23.957370 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:23.957342    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:39:29.194094 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.194058    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"]
Jul 23 18:39:29.197635 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.197613    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:29.200006 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.199982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:39:29.205216 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.205192    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"]
Jul 23 18:39:29.291861 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.291812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be40d4db-65f6-4c7b-a929-f43acb961f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22682a-mhblc\" (UID: \"be40d4db-65f6-4c7b-a929-f43acb961f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:29.393111 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.393080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be40d4db-65f6-4c7b-a929-f43acb961f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22682a-mhblc\" (UID: \"be40d4db-65f6-4c7b-a929-f43acb961f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:29.393663 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.393644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be40d4db-65f6-4c7b-a929-f43acb961f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22682a-mhblc\" (UID: \"be40d4db-65f6-4c7b-a929-f43acb961f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:29.511541 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.511508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:29.654760 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:29.654732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"]
Jul 23 18:39:29.656350 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:39:29.656320    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe40d4db_65f6_4c7b_a929_f43acb961f30.slice/crio-9cd8938efb51ba40a51a4ba99bb498a1627cc1bf9102b7965dc938457655ad57 WatchSource:0}: Error finding container 9cd8938efb51ba40a51a4ba99bb498a1627cc1bf9102b7965dc938457655ad57: Status 404 returned error can't find the container with id 9cd8938efb51ba40a51a4ba99bb498a1627cc1bf9102b7965dc938457655ad57
Jul 23 18:39:30.013565 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:30.013531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc" event={"ID":"be40d4db-65f6-4c7b-a929-f43acb961f30","Type":"ContainerStarted","Data":"ea386e9a81642295a9dcdffef3d52a2c128d22aeee78f3c3d46b94b9ff95c146"}
Jul 23 18:39:30.013725 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:30.013581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc" event={"ID":"be40d4db-65f6-4c7b-a929-f43acb961f30","Type":"ContainerStarted","Data":"9cd8938efb51ba40a51a4ba99bb498a1627cc1bf9102b7965dc938457655ad57"}
Jul 23 18:39:30.028542 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:30.028479    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc" podStartSLOduration=1.028462068 podStartE2EDuration="1.028462068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:30.02800633 +0000 UTC m=+2408.818163459" watchObservedRunningTime="2026-07-23 18:39:30.028462068 +0000 UTC m=+2408.818619201"
Jul 23 18:39:31.021091 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:31.021050    2562 generic.go:356] "Generic (PLEG): container finished" podID="be40d4db-65f6-4c7b-a929-f43acb961f30" containerID="ea386e9a81642295a9dcdffef3d52a2c128d22aeee78f3c3d46b94b9ff95c146" exitCode=0
Jul 23 18:39:31.021552 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:31.021151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc" event={"ID":"be40d4db-65f6-4c7b-a929-f43acb961f30","Type":"ContainerDied","Data":"ea386e9a81642295a9dcdffef3d52a2c128d22aeee78f3c3d46b94b9ff95c146"}
Jul 23 18:39:32.163737 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:32.163712    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:32.320842 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:32.320759    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be40d4db-65f6-4c7b-a929-f43acb961f30-maas-api-service-ca\") pod \"be40d4db-65f6-4c7b-a929-f43acb961f30\" (UID: \"be40d4db-65f6-4c7b-a929-f43acb961f30\") "
Jul 23 18:39:32.321172 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:32.321153    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be40d4db-65f6-4c7b-a929-f43acb961f30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be40d4db-65f6-4c7b-a929-f43acb961f30" (UID: "be40d4db-65f6-4c7b-a929-f43acb961f30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:39:32.422256 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:32.422224    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be40d4db-65f6-4c7b-a929-f43acb961f30-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:39:33.033774 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:33.033747    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"
Jul 23 18:39:33.033922 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:33.033747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc" event={"ID":"be40d4db-65f6-4c7b-a929-f43acb961f30","Type":"ContainerDied","Data":"9cd8938efb51ba40a51a4ba99bb498a1627cc1bf9102b7965dc938457655ad57"}
Jul 23 18:39:33.033922 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:33.033867    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cd8938efb51ba40a51a4ba99bb498a1627cc1bf9102b7965dc938457655ad57"
Jul 23 18:39:44.993725 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:44.993515    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"]
Jul 23 18:39:44.994297 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:44.993770    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m" podUID="8c521241-546a-4a35-a9f1-a712a0a2bbef" containerName="maas-api" containerID="cri-o://dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f" gracePeriod=30
Jul 23 18:39:45.244438 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.244378    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:45.340186 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.340136    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c521241-546a-4a35-a9f1-a712a0a2bbef-maas-api-tls\") pod \"8c521241-546a-4a35-a9f1-a712a0a2bbef\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") "
Jul 23 18:39:45.340365 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.340201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5m4h\" (UniqueName: \"kubernetes.io/projected/8c521241-546a-4a35-a9f1-a712a0a2bbef-kube-api-access-c5m4h\") pod \"8c521241-546a-4a35-a9f1-a712a0a2bbef\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") "
Jul 23 18:39:45.340365 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.340282    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c521241-546a-4a35-a9f1-a712a0a2bbef-openshift-service-ca\") pod \"8c521241-546a-4a35-a9f1-a712a0a2bbef\" (UID: \"8c521241-546a-4a35-a9f1-a712a0a2bbef\") "
Jul 23 18:39:45.340621 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.340593    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c521241-546a-4a35-a9f1-a712a0a2bbef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c521241-546a-4a35-a9f1-a712a0a2bbef" (UID: "8c521241-546a-4a35-a9f1-a712a0a2bbef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:39:45.342366 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.342340    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c521241-546a-4a35-a9f1-a712a0a2bbef-kube-api-access-c5m4h" (OuterVolumeSpecName: "kube-api-access-c5m4h") pod "8c521241-546a-4a35-a9f1-a712a0a2bbef" (UID: "8c521241-546a-4a35-a9f1-a712a0a2bbef"). InnerVolumeSpecName "kube-api-access-c5m4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:39:45.342472 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.342386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c521241-546a-4a35-a9f1-a712a0a2bbef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c521241-546a-4a35-a9f1-a712a0a2bbef" (UID: "8c521241-546a-4a35-a9f1-a712a0a2bbef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:39:45.441047 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.440986    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c521241-546a-4a35-a9f1-a712a0a2bbef-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:39:45.441047 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.441047    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5m4h\" (UniqueName: \"kubernetes.io/projected/8c521241-546a-4a35-a9f1-a712a0a2bbef-kube-api-access-c5m4h\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:39:45.441047 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:45.441058    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c521241-546a-4a35-a9f1-a712a0a2bbef-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:39:46.106777 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.106741    2562 generic.go:356] "Generic (PLEG): container finished" podID="8c521241-546a-4a35-a9f1-a712a0a2bbef" containerID="dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f" exitCode=0
Jul 23 18:39:46.107279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.106792    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"
Jul 23 18:39:46.107279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.106829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m" event={"ID":"8c521241-546a-4a35-a9f1-a712a0a2bbef","Type":"ContainerDied","Data":"dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f"}
Jul 23 18:39:46.107279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.106882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m" event={"ID":"8c521241-546a-4a35-a9f1-a712a0a2bbef","Type":"ContainerDied","Data":"9fc4611310edda22867f0acde48f0c4d6f7ebc90855edbb669eee168f512924d"}
Jul 23 18:39:46.107279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.106903    2562 scope.go:117] "RemoveContainer" containerID="dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f"
Jul 23 18:39:46.115867 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.115850    2562 scope.go:117] "RemoveContainer" containerID="dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f"
Jul 23 18:39:46.116139 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:39:46.116123    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f\": container with ID starting with dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f not found: ID does not exist" containerID="dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f"
Jul 23 18:39:46.116196 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.116146    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f"} err="failed to get container status \"dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f\": rpc error: code = NotFound desc = could not find container \"dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f\": container with ID starting with dd3dc144c3d41f70492dd877f6ab9de262bba7d17ed2d90a3c1be718abed9f4f not found: ID does not exist"
Jul 23 18:39:46.124033 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.124009    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"]
Jul 23 18:39:46.130347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:46.130326    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22682a-7ddd755f5-kcj2m"]
Jul 23 18:39:47.609648 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:47.609613    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c521241-546a-4a35-a9f1-a712a0a2bbef" path="/var/lib/kubelet/pods/8c521241-546a-4a35-a9f1-a712a0a2bbef/volumes"
Jul 23 18:39:49.280048 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:49.280004    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"]
Jul 23 18:39:49.284180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:49.284151    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22682a-mhblc"]
Jul 23 18:39:49.608241 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:49.608157    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be40d4db-65f6-4c7b-a929-f43acb961f30" path="/var/lib/kubelet/pods/be40d4db-65f6-4c7b-a929-f43acb961f30/volumes"
Jul 23 18:39:56.450677 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.450643    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"]
Jul 23 18:39:56.451069 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.451057    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be40d4db-65f6-4c7b-a929-f43acb961f30" containerName="revoke-keys"
Jul 23 18:39:56.451112 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.451071    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be40d4db-65f6-4c7b-a929-f43acb961f30" containerName="revoke-keys"
Jul 23 18:39:56.451112 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.451085    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c521241-546a-4a35-a9f1-a712a0a2bbef" containerName="maas-api"
Jul 23 18:39:56.451112 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.451091    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c521241-546a-4a35-a9f1-a712a0a2bbef" containerName="maas-api"
Jul 23 18:39:56.451210 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.451146    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be40d4db-65f6-4c7b-a929-f43acb961f30" containerName="revoke-keys"
Jul 23 18:39:56.451210 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.451153    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c521241-546a-4a35-a9f1-a712a0a2bbef" containerName="maas-api"
Jul 23 18:39:56.455729 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.455708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:39:56.458106 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.458075    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:39:56.464156 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.464132    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"]
Jul 23 18:39:56.539016 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.538979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18212ef-397a-44e4-a73a-8c14419fe07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f77b55-xs47j\" (UID: \"e18212ef-397a-44e4-a73a-8c14419fe07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:39:56.640444 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.640409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18212ef-397a-44e4-a73a-8c14419fe07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f77b55-xs47j\" (UID: \"e18212ef-397a-44e4-a73a-8c14419fe07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:39:56.641022 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.641004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18212ef-397a-44e4-a73a-8c14419fe07c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f77b55-xs47j\" (UID: \"e18212ef-397a-44e4-a73a-8c14419fe07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:39:56.768304 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.768262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:39:56.914316 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:56.914287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"]
Jul 23 18:39:56.916685 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:39:56.916653    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18212ef_397a_44e4_a73a_8c14419fe07c.slice/crio-2740edf04107890c3dc6f0304e000871f8707ebc44e5ed2cccebaa16d14e7aee WatchSource:0}: Error finding container 2740edf04107890c3dc6f0304e000871f8707ebc44e5ed2cccebaa16d14e7aee: Status 404 returned error can't find the container with id 2740edf04107890c3dc6f0304e000871f8707ebc44e5ed2cccebaa16d14e7aee
Jul 23 18:39:57.175218 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:57.175181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j" event={"ID":"e18212ef-397a-44e4-a73a-8c14419fe07c","Type":"ContainerStarted","Data":"2740edf04107890c3dc6f0304e000871f8707ebc44e5ed2cccebaa16d14e7aee"}
Jul 23 18:39:58.182219 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:58.182133    2562 generic.go:356] "Generic (PLEG): container finished" podID="e18212ef-397a-44e4-a73a-8c14419fe07c" containerID="00edf5786ca0d88671af5a4c840224fbb60c2296c6af2f8d19c29155a88b8fcc" exitCode=0
Jul 23 18:39:58.182595 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:58.182219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j" event={"ID":"e18212ef-397a-44e4-a73a-8c14419fe07c","Type":"ContainerDied","Data":"00edf5786ca0d88671af5a4c840224fbb60c2296c6af2f8d19c29155a88b8fcc"}
Jul 23 18:39:59.323385 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:59.323358    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:39:59.464791 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:59.464693    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18212ef-397a-44e4-a73a-8c14419fe07c-maas-api-service-ca\") pod \"e18212ef-397a-44e4-a73a-8c14419fe07c\" (UID: \"e18212ef-397a-44e4-a73a-8c14419fe07c\") "
Jul 23 18:39:59.465148 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:59.465115    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18212ef-397a-44e4-a73a-8c14419fe07c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e18212ef-397a-44e4-a73a-8c14419fe07c" (UID: "e18212ef-397a-44e4-a73a-8c14419fe07c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:39:59.565690 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:39:59.565653    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18212ef-397a-44e4-a73a-8c14419fe07c-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:40:00.195110 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:00.195085    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"
Jul 23 18:40:00.195324 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:00.195076    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j" event={"ID":"e18212ef-397a-44e4-a73a-8c14419fe07c","Type":"ContainerDied","Data":"2740edf04107890c3dc6f0304e000871f8707ebc44e5ed2cccebaa16d14e7aee"}
Jul 23 18:40:00.195324 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:00.195219    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2740edf04107890c3dc6f0304e000871f8707ebc44e5ed2cccebaa16d14e7aee"
Jul 23 18:40:11.944615 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:11.944577    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"]
Jul 23 18:40:11.945172 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:11.944829    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5" podUID="290b274d-c871-452d-b90f-304c54f09ad4" containerName="maas-api" containerID="cri-o://d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0" gracePeriod=30
Jul 23 18:40:12.204471 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.204403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:40:12.262094 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.262065    2562 generic.go:356] "Generic (PLEG): container finished" podID="290b274d-c871-452d-b90f-304c54f09ad4" containerID="d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0" exitCode=0
Jul 23 18:40:12.262287 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.262124    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"
Jul 23 18:40:12.262287 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.262146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5" event={"ID":"290b274d-c871-452d-b90f-304c54f09ad4","Type":"ContainerDied","Data":"d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0"}
Jul 23 18:40:12.262287 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.262193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5" event={"ID":"290b274d-c871-452d-b90f-304c54f09ad4","Type":"ContainerDied","Data":"8bb7698b3130a842f204333f9a6689cafefe39678fed6d191b0c7f08baa542d0"}
Jul 23 18:40:12.262287 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.262228    2562 scope.go:117] "RemoveContainer" containerID="d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0"
Jul 23 18:40:12.271458 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.271443    2562 scope.go:117] "RemoveContainer" containerID="d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0"
Jul 23 18:40:12.271682 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:40:12.271666    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0\": container with ID starting with d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0 not found: ID does not exist" containerID="d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0"
Jul 23 18:40:12.271720 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.271691    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0"} err="failed to get container status \"d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0\": rpc error: code = NotFound desc = could not find container \"d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0\": container with ID starting with d61a9c72a651c9d76d9745c6d852b1c043e00f68c48957a09c1c6806a72a69f0 not found: ID does not exist"
Jul 23 18:40:12.384486 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.384450    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b274d-c871-452d-b90f-304c54f09ad4-openshift-service-ca\") pod \"290b274d-c871-452d-b90f-304c54f09ad4\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") "
Jul 23 18:40:12.384672 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.384516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290b274d-c871-452d-b90f-304c54f09ad4-maas-api-tls\") pod \"290b274d-c871-452d-b90f-304c54f09ad4\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") "
Jul 23 18:40:12.384672 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.384569    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7fs5\" (UniqueName: \"kubernetes.io/projected/290b274d-c871-452d-b90f-304c54f09ad4-kube-api-access-f7fs5\") pod \"290b274d-c871-452d-b90f-304c54f09ad4\" (UID: \"290b274d-c871-452d-b90f-304c54f09ad4\") "
Jul 23 18:40:12.384882 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.384857    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290b274d-c871-452d-b90f-304c54f09ad4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "290b274d-c871-452d-b90f-304c54f09ad4" (UID: "290b274d-c871-452d-b90f-304c54f09ad4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:40:12.386761 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.386735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290b274d-c871-452d-b90f-304c54f09ad4-kube-api-access-f7fs5" (OuterVolumeSpecName: "kube-api-access-f7fs5") pod "290b274d-c871-452d-b90f-304c54f09ad4" (UID: "290b274d-c871-452d-b90f-304c54f09ad4"). InnerVolumeSpecName "kube-api-access-f7fs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:40:12.386918 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.386903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290b274d-c871-452d-b90f-304c54f09ad4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "290b274d-c871-452d-b90f-304c54f09ad4" (UID: "290b274d-c871-452d-b90f-304c54f09ad4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:40:12.485428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.485327    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b274d-c871-452d-b90f-304c54f09ad4-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:40:12.485428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.485374    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290b274d-c871-452d-b90f-304c54f09ad4-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:40:12.485428 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.485386    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7fs5\" (UniqueName: \"kubernetes.io/projected/290b274d-c871-452d-b90f-304c54f09ad4-kube-api-access-f7fs5\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:40:12.584764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.584732    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"]
Jul 23 18:40:12.588226 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:12.588200    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f77b55-6cb88c98cf-xf9k5"]
Jul 23 18:40:13.609315 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:13.609283    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290b274d-c871-452d-b90f-304c54f09ad4" path="/var/lib/kubelet/pods/290b274d-c871-452d-b90f-304c54f09ad4/volumes"
Jul 23 18:40:16.541121 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:16.541081    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"]
Jul 23 18:40:16.544551 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:16.544516    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f77b55-xs47j"]
Jul 23 18:40:17.609374 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:17.609336    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e18212ef-397a-44e4-a73a-8c14419fe07c" path="/var/lib/kubelet/pods/e18212ef-397a-44e4-a73a-8c14419fe07c/volumes"
Jul 23 18:40:28.059013 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.058978    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"]
Jul 23 18:40:28.059461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.059417    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e18212ef-397a-44e4-a73a-8c14419fe07c" containerName="revoke-keys"
Jul 23 18:40:28.059461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.059435    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e18212ef-397a-44e4-a73a-8c14419fe07c" containerName="revoke-keys"
Jul 23 18:40:28.059461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.059451    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290b274d-c871-452d-b90f-304c54f09ad4" containerName="maas-api"
Jul 23 18:40:28.059461 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.059460    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="290b274d-c871-452d-b90f-304c54f09ad4" containerName="maas-api"
Jul 23 18:40:28.059666 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.059588    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="290b274d-c871-452d-b90f-304c54f09ad4" containerName="maas-api"
Jul 23 18:40:28.059666 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.059608    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e18212ef-397a-44e4-a73a-8c14419fe07c" containerName="revoke-keys"
Jul 23 18:40:28.062759 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.062740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.065101 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.065074    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5f06b1-serving-cert\""
Jul 23 18:40:28.065248 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.065127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p7r4p\""
Jul 23 18:40:28.074210 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.074183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"]
Jul 23 18:40:28.116998 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.116930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6hn\" (UniqueName: \"kubernetes.io/projected/1eb012b0-432e-429b-a0f1-ce65e32350ff-kube-api-access-dn6hn\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.117178 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.117036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb012b0-432e-429b-a0f1-ce65e32350ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.117178 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.117063    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb012b0-432e-429b-a0f1-ce65e32350ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.217846 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.217804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6hn\" (UniqueName: \"kubernetes.io/projected/1eb012b0-432e-429b-a0f1-ce65e32350ff-kube-api-access-dn6hn\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.218056 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.217868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb012b0-432e-429b-a0f1-ce65e32350ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.218056 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.217891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb012b0-432e-429b-a0f1-ce65e32350ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.218543 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.218518    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb012b0-432e-429b-a0f1-ce65e32350ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.220443 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.220421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb012b0-432e-429b-a0f1-ce65e32350ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.228236 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.228211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6hn\" (UniqueName: \"kubernetes.io/projected/1eb012b0-432e-429b-a0f1-ce65e32350ff-kube-api-access-dn6hn\") pod \"maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.373661 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.373573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:28.527868 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:28.527836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"]
Jul 23 18:40:28.529673 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:40:28.529646    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb012b0_432e_429b_a0f1_ce65e32350ff.slice/crio-76c766473865f0101e759e08d6826a4b0fb03b96efa32a61a7d24ef6f7333164 WatchSource:0}: Error finding container 76c766473865f0101e759e08d6826a4b0fb03b96efa32a61a7d24ef6f7333164: Status 404 returned error can't find the container with id 76c766473865f0101e759e08d6826a4b0fb03b96efa32a61a7d24ef6f7333164
Jul 23 18:40:29.366066 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:29.366026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj" event={"ID":"1eb012b0-432e-429b-a0f1-ce65e32350ff","Type":"ContainerStarted","Data":"c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7"}
Jul 23 18:40:29.366567 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:29.366087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj" event={"ID":"1eb012b0-432e-429b-a0f1-ce65e32350ff","Type":"ContainerStarted","Data":"76c766473865f0101e759e08d6826a4b0fb03b96efa32a61a7d24ef6f7333164"}
Jul 23 18:40:29.366567 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:29.366166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:29.385686 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:29.385636    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj" podStartSLOduration=0.977344856 podStartE2EDuration="1.38562254s" totalImagesPullingTime="408.277684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:40:29.384134972 +0000 UTC m=+2468.174292087" watchObservedRunningTime="2026-07-23 18:40:29.38562254 +0000 UTC m=+2468.175779655"
Jul 23 18:40:35.376997 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.376967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:40:35.959041 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.959001    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"]
Jul 23 18:40:35.963439 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.963415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:35.966176 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.966146    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-8ee985-serving-cert\""
Jul 23 18:40:35.973533 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.973507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"]
Jul 23 18:40:35.983502 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.983479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87eb14c-921a-45df-8ec7-b9f0f8a04588-maas-api-tls\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:35.983618 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.983511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87eb14c-921a-45df-8ec7-b9f0f8a04588-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:35.983618 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:35.983535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2b22\" (UniqueName: \"kubernetes.io/projected/b87eb14c-921a-45df-8ec7-b9f0f8a04588-kube-api-access-s2b22\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.084551 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.084511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87eb14c-921a-45df-8ec7-b9f0f8a04588-maas-api-tls\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.084750 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.084554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87eb14c-921a-45df-8ec7-b9f0f8a04588-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.084750 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.084683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2b22\" (UniqueName: \"kubernetes.io/projected/b87eb14c-921a-45df-8ec7-b9f0f8a04588-kube-api-access-s2b22\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.085240 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.085217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87eb14c-921a-45df-8ec7-b9f0f8a04588-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.087203 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.087183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87eb14c-921a-45df-8ec7-b9f0f8a04588-maas-api-tls\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.093600 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.093580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2b22\" (UniqueName: \"kubernetes.io/projected/b87eb14c-921a-45df-8ec7-b9f0f8a04588-kube-api-access-s2b22\") pod \"maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.275301 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.275267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:36.416026 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:36.415994    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"]
Jul 23 18:40:36.417622 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:40:36.417597    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87eb14c_921a_45df_8ec7_b9f0f8a04588.slice/crio-50f622a4158450d1242da99469d2fd752260af5cf6559da8004b837c22035e7f WatchSource:0}: Error finding container 50f622a4158450d1242da99469d2fd752260af5cf6559da8004b837c22035e7f: Status 404 returned error can't find the container with id 50f622a4158450d1242da99469d2fd752260af5cf6559da8004b837c22035e7f
Jul 23 18:40:37.416055 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:37.416020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw" event={"ID":"b87eb14c-921a-45df-8ec7-b9f0f8a04588","Type":"ContainerStarted","Data":"7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9"}
Jul 23 18:40:37.416451 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:37.416071    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw" event={"ID":"b87eb14c-921a-45df-8ec7-b9f0f8a04588","Type":"ContainerStarted","Data":"50f622a4158450d1242da99469d2fd752260af5cf6559da8004b837c22035e7f"}
Jul 23 18:40:37.416451 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:37.416157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:40:37.437970 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:37.437892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw" podStartSLOduration=1.968021864 podStartE2EDuration="2.437877879s" totalImagesPullingTime="469.856015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:40:37.435740505 +0000 UTC m=+2476.225897619" watchObservedRunningTime="2026-07-23 18:40:37.437877879 +0000 UTC m=+2476.228034992"
Jul 23 18:40:43.427352 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:40:43.427302    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:44:21.828572 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:44:21.828499    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:44:21.841094 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:44:21.841071    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:44:21.945249 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:44:21.945219    2562 scope.go:117] "RemoveContainer" containerID="df2f5e1205f6e4770361d2bb443de0509ea5ab17cb794d1566fb7e8ba3f53a01"
Jul 23 18:45:00.157270 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.157238    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"]
Jul 23 18:45:00.161099 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.161068    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"]
Jul 23 18:45:00.161292 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.161264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"
Jul 23 18:45:00.163852 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.163798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:45:00.164872 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.164851    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"]
Jul 23 18:45:00.165122 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.165101    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"
Jul 23 18:45:00.168469 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.168449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"]
Jul 23 18:45:00.168579 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.168521    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"
Jul 23 18:45:00.174743 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.174724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"
Jul 23 18:45:00.179610 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.179581    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"]
Jul 23 18:45:00.185932 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.185909    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"
Jul 23 18:45:00.187199 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.187173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"]
Jul 23 18:45:00.225298 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.225275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"
Jul 23 18:45:00.362000 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.361884    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"]
Jul 23 18:45:00.362255 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:45:00.362213    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74225b3_0209_47d6_a784_ac0656e6b09d.slice/crio-211dfd955234c4164ceee07b69bf4a52a9243393395f6182eaaecc63a8390da6 WatchSource:0}: Error finding container 211dfd955234c4164ceee07b69bf4a52a9243393395f6182eaaecc63a8390da6: Status 404 returned error can't find the container with id 211dfd955234c4164ceee07b69bf4a52a9243393395f6182eaaecc63a8390da6
Jul 23 18:45:00.419185 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.419159    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"]
Jul 23 18:45:00.421303 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:45:00.421272    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05ff318_b364_4d32_a039_0209dfd5732f.slice/crio-764a9c5330fea4e8318a2972dfece80bfd9e6eec2d526494cdfc60cbe4b07c8f WatchSource:0}: Error finding container 764a9c5330fea4e8318a2972dfece80bfd9e6eec2d526494cdfc60cbe4b07c8f: Status 404 returned error can't find the container with id 764a9c5330fea4e8318a2972dfece80bfd9e6eec2d526494cdfc60cbe4b07c8f
Jul 23 18:45:00.595579 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.595539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"]
Jul 23 18:45:00.598050 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:45:00.598021    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f1b63_aa99_4059_9902_b85c21cb24df.slice/crio-894bc73ff1198a2d5cb89afb9ac1a24e3545720022e9c00ea064dc5f0e838100 WatchSource:0}: Error finding container 894bc73ff1198a2d5cb89afb9ac1a24e3545720022e9c00ea064dc5f0e838100: Status 404 returned error can't find the container with id 894bc73ff1198a2d5cb89afb9ac1a24e3545720022e9c00ea064dc5f0e838100
Jul 23 18:45:00.951505 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.951469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2" event={"ID":"b74225b3-0209-47d6-a784-ac0656e6b09d","Type":"ContainerStarted","Data":"85cfd8469cecf96e8f9818f9ea62aa5fc7ec8e69d6fe9695815e5486291abb41"}
Jul 23 18:45:00.951692 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.951526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2" event={"ID":"b74225b3-0209-47d6-a784-ac0656e6b09d","Type":"ContainerStarted","Data":"211dfd955234c4164ceee07b69bf4a52a9243393395f6182eaaecc63a8390da6"}
Jul 23 18:45:00.952779 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.952747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt" event={"ID":"a39f1b63-aa99-4059-9902-b85c21cb24df","Type":"ContainerStarted","Data":"894bc73ff1198a2d5cb89afb9ac1a24e3545720022e9c00ea064dc5f0e838100"}
Jul 23 18:45:00.954501 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.954463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz" event={"ID":"f05ff318-b364-4d32-a039-0209dfd5732f","Type":"ContainerStarted","Data":"e72bf062825365204068f3a0f910bbc48a572081519e6e87475c3ddf826b70a1"}
Jul 23 18:45:00.954621 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.954526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz" event={"ID":"f05ff318-b364-4d32-a039-0209dfd5732f","Type":"ContainerStarted","Data":"764a9c5330fea4e8318a2972dfece80bfd9e6eec2d526494cdfc60cbe4b07c8f"}
Jul 23 18:45:00.976225 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:00.976139    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2" podStartSLOduration=0.976125184 podStartE2EDuration="976.125184ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:00.97475996 +0000 UTC m=+2739.764917075" watchObservedRunningTime="2026-07-23 18:45:00.976125184 +0000 UTC m=+2739.766282299"
Jul 23 18:45:01.004226 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.004174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz" podStartSLOduration=1.004156721 podStartE2EDuration="1.004156721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:01.001038502 +0000 UTC m=+2739.791195638" watchObservedRunningTime="2026-07-23 18:45:01.004156721 +0000 UTC m=+2739.794313835"
Jul 23 18:45:01.961803 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.961767    2562 generic.go:356] "Generic (PLEG): container finished" podID="a39f1b63-aa99-4059-9902-b85c21cb24df" containerID="495719c60fe05ff5270b2b8407416c30ab6297e6b3fc1ec634f91c1acf53fd93" exitCode=0
Jul 23 18:45:01.962248 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.961865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt" event={"ID":"a39f1b63-aa99-4059-9902-b85c21cb24df","Type":"ContainerDied","Data":"495719c60fe05ff5270b2b8407416c30ab6297e6b3fc1ec634f91c1acf53fd93"}
Jul 23 18:45:01.963491 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.963473    2562 generic.go:356] "Generic (PLEG): container finished" podID="f05ff318-b364-4d32-a039-0209dfd5732f" containerID="e72bf062825365204068f3a0f910bbc48a572081519e6e87475c3ddf826b70a1" exitCode=0
Jul 23 18:45:01.963584 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.963502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz" event={"ID":"f05ff318-b364-4d32-a039-0209dfd5732f","Type":"ContainerDied","Data":"e72bf062825365204068f3a0f910bbc48a572081519e6e87475c3ddf826b70a1"}
Jul 23 18:45:01.964977 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.964929    2562 generic.go:356] "Generic (PLEG): container finished" podID="b74225b3-0209-47d6-a784-ac0656e6b09d" containerID="85cfd8469cecf96e8f9818f9ea62aa5fc7ec8e69d6fe9695815e5486291abb41" exitCode=0
Jul 23 18:45:01.965072 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:01.965031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2" event={"ID":"b74225b3-0209-47d6-a784-ac0656e6b09d","Type":"ContainerDied","Data":"85cfd8469cecf96e8f9818f9ea62aa5fc7ec8e69d6fe9695815e5486291abb41"}
Jul 23 18:45:03.107782 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.107748    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"
Jul 23 18:45:03.154573 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.154541    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"
Jul 23 18:45:03.185651 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.185621    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"
Jul 23 18:45:03.977744 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.977653    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"
Jul 23 18:45:03.977744 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.977668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt" event={"ID":"a39f1b63-aa99-4059-9902-b85c21cb24df","Type":"ContainerDied","Data":"894bc73ff1198a2d5cb89afb9ac1a24e3545720022e9c00ea064dc5f0e838100"}
Jul 23 18:45:03.977744 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.977719    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="894bc73ff1198a2d5cb89afb9ac1a24e3545720022e9c00ea064dc5f0e838100"
Jul 23 18:45:03.979186 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.979160    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"
Jul 23 18:45:03.979347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.979207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz" event={"ID":"f05ff318-b364-4d32-a039-0209dfd5732f","Type":"ContainerDied","Data":"764a9c5330fea4e8318a2972dfece80bfd9e6eec2d526494cdfc60cbe4b07c8f"}
Jul 23 18:45:03.979347 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.979252    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764a9c5330fea4e8318a2972dfece80bfd9e6eec2d526494cdfc60cbe4b07c8f"
Jul 23 18:45:03.980694 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.980675    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2"
Jul 23 18:45:03.980800 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.980688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-6lzq2" event={"ID":"b74225b3-0209-47d6-a784-ac0656e6b09d","Type":"ContainerDied","Data":"211dfd955234c4164ceee07b69bf4a52a9243393395f6182eaaecc63a8390da6"}
Jul 23 18:45:03.980800 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:03.980733    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="211dfd955234c4164ceee07b69bf4a52a9243393395f6182eaaecc63a8390da6"
Jul 23 18:45:21.973322 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:45:21.973289    2562 scope.go:117] "RemoveContainer" containerID="29ed3d30503b2d49d7f67d7044db84976422e9e2f08a77b4379392d7aea59261"
Jul 23 18:46:21.995023 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:46:21.994989    2562 scope.go:117] "RemoveContainer" containerID="ea386e9a81642295a9dcdffef3d52a2c128d22aeee78f3c3d46b94b9ff95c146"
Jul 23 18:46:22.004013 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:46:22.003994    2562 scope.go:117] "RemoveContainer" containerID="00edf5786ca0d88671af5a4c840224fbb60c2296c6af2f8d19c29155a88b8fcc"
Jul 23 18:47:52.011539 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.011503    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-140622583912112"]
Jul 23 18:47:52.012053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.011934    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39f1b63-aa99-4059-9902-b85c21cb24df" containerName="cleanup"
Jul 23 18:47:52.012053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.011962    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39f1b63-aa99-4059-9902-b85c21cb24df" containerName="cleanup"
Jul 23 18:47:52.012053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.011978    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74225b3-0209-47d6-a784-ac0656e6b09d" containerName="cleanup"
Jul 23 18:47:52.012053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.011987    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74225b3-0209-47d6-a784-ac0656e6b09d" containerName="cleanup"
Jul 23 18:47:52.012053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.012004    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f05ff318-b364-4d32-a039-0209dfd5732f" containerName="cleanup"
Jul 23 18:47:52.012053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.012010    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f05ff318-b364-4d32-a039-0209dfd5732f" containerName="cleanup"
Jul 23 18:47:52.012286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.012082    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f05ff318-b364-4d32-a039-0209dfd5732f" containerName="cleanup"
Jul 23 18:47:52.012286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.012091    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39f1b63-aa99-4059-9902-b85c21cb24df" containerName="cleanup"
Jul 23 18:47:52.012286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.012099    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74225b3-0209-47d6-a784-ac0656e6b09d" containerName="cleanup"
Jul 23 18:47:52.015286 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.015267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:52.022698 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.022398    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qkcfk\""
Jul 23 18:47:52.026198 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.026172    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583912112"]
Jul 23 18:47:52.037180 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.037152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dhl\" (UniqueName: \"kubernetes.io/projected/aedad270-3108-4271-a154-319bd896bfba-kube-api-access-k4dhl\") pod \"test-curl-7426-140622583912112\" (UID: \"aedad270-3108-4271-a154-319bd896bfba\") " pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:52.138500 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.138461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dhl\" (UniqueName: \"kubernetes.io/projected/aedad270-3108-4271-a154-319bd896bfba-kube-api-access-k4dhl\") pod \"test-curl-7426-140622583912112\" (UID: \"aedad270-3108-4271-a154-319bd896bfba\") " pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:52.149556 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.149529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dhl\" (UniqueName: \"kubernetes.io/projected/aedad270-3108-4271-a154-319bd896bfba-kube-api-access-k4dhl\") pod \"test-curl-7426-140622583912112\" (UID: \"aedad270-3108-4271-a154-319bd896bfba\") " pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:52.331516 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.331477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:52.480482 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.480451    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583912112"]
Jul 23 18:47:52.481856 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:47:52.481826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaedad270_3108_4271_a154_319bd896bfba.slice/crio-85e93968ae20a1438fb2f9cc0084c9a062ab202c14ff41369ca262b06db58395 WatchSource:0}: Error finding container 85e93968ae20a1438fb2f9cc0084c9a062ab202c14ff41369ca262b06db58395: Status 404 returned error can't find the container with id 85e93968ae20a1438fb2f9cc0084c9a062ab202c14ff41369ca262b06db58395
Jul 23 18:47:52.483248 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.483229    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:47:52.964561 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:52.964456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583912112" event={"ID":"aedad270-3108-4271-a154-319bd896bfba","Type":"ContainerStarted","Data":"85e93968ae20a1438fb2f9cc0084c9a062ab202c14ff41369ca262b06db58395"}
Jul 23 18:47:53.972421 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:53.972384    2562 generic.go:356] "Generic (PLEG): container finished" podID="aedad270-3108-4271-a154-319bd896bfba" containerID="2e03fa6ab734d873bd3d6817a799d0c6bdc4591baaa967e2f9a78fe0e36efc47" exitCode=0
Jul 23 18:47:53.972806 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:53.972454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583912112" event={"ID":"aedad270-3108-4271-a154-319bd896bfba","Type":"ContainerDied","Data":"2e03fa6ab734d873bd3d6817a799d0c6bdc4591baaa967e2f9a78fe0e36efc47"}
Jul 23 18:47:55.126905 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.126877    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:55.164593 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.164564    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7426-140622583912112_aedad270-3108-4271-a154-319bd896bfba/test-curl-7426-140622583912112/0.log"
Jul 23 18:47:55.168763 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.168737    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4dhl\" (UniqueName: \"kubernetes.io/projected/aedad270-3108-4271-a154-319bd896bfba-kube-api-access-k4dhl\") pod \"aedad270-3108-4271-a154-319bd896bfba\" (UID: \"aedad270-3108-4271-a154-319bd896bfba\") "
Jul 23 18:47:55.171057 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.171030    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedad270-3108-4271-a154-319bd896bfba-kube-api-access-k4dhl" (OuterVolumeSpecName: "kube-api-access-k4dhl") pod "aedad270-3108-4271-a154-319bd896bfba" (UID: "aedad270-3108-4271-a154-319bd896bfba"). InnerVolumeSpecName "kube-api-access-k4dhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:47:55.190829 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.190797    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-140622583912112"]
Jul 23 18:47:55.194442 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.194404    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-140622583912112"]
Jul 23 18:47:55.269563 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.269505    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dhl\" (UniqueName: \"kubernetes.io/projected/aedad270-3108-4271-a154-319bd896bfba-kube-api-access-k4dhl\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:47:55.329184 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.329145    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-140622583911856"]
Jul 23 18:47:55.329585 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.329562    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aedad270-3108-4271-a154-319bd896bfba" containerName="test-curl-7426-140622583912112"
Jul 23 18:47:55.329585 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.329581    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aedad270-3108-4271-a154-319bd896bfba" containerName="test-curl-7426-140622583912112"
Jul 23 18:47:55.329666 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.329662    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aedad270-3108-4271-a154-319bd896bfba" containerName="test-curl-7426-140622583912112"
Jul 23 18:47:55.332803 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.332786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:55.340929 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.340903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583911856"]
Jul 23 18:47:55.370419 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.370383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sshqs\" (UniqueName: \"kubernetes.io/projected/9212648b-6565-477e-8b36-fab63074c683-kube-api-access-sshqs\") pod \"test-curl-7426-140622583911856\" (UID: \"9212648b-6565-477e-8b36-fab63074c683\") " pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:55.471212 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.471177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sshqs\" (UniqueName: \"kubernetes.io/projected/9212648b-6565-477e-8b36-fab63074c683-kube-api-access-sshqs\") pod \"test-curl-7426-140622583911856\" (UID: \"9212648b-6565-477e-8b36-fab63074c683\") " pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:55.479573 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.479540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sshqs\" (UniqueName: \"kubernetes.io/projected/9212648b-6565-477e-8b36-fab63074c683-kube-api-access-sshqs\") pod \"test-curl-7426-140622583911856\" (UID: \"9212648b-6565-477e-8b36-fab63074c683\") " pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:55.610097 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.609977    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aedad270-3108-4271-a154-319bd896bfba" path="/var/lib/kubelet/pods/aedad270-3108-4271-a154-319bd896bfba/volumes"
Jul 23 18:47:55.642779 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.642734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:55.789184 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.789157    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583911856"]
Jul 23 18:47:55.790861 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:47:55.790834    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9212648b_6565_477e_8b36_fab63074c683.slice/crio-5517712bc180c804073a626ec83f6d254745a203ea30223e2f6ffceeb4e829b4 WatchSource:0}: Error finding container 5517712bc180c804073a626ec83f6d254745a203ea30223e2f6ffceeb4e829b4: Status 404 returned error can't find the container with id 5517712bc180c804073a626ec83f6d254745a203ea30223e2f6ffceeb4e829b4
Jul 23 18:47:55.985785 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.985690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583911856" event={"ID":"9212648b-6565-477e-8b36-fab63074c683","Type":"ContainerStarted","Data":"5517712bc180c804073a626ec83f6d254745a203ea30223e2f6ffceeb4e829b4"}
Jul 23 18:47:55.987122 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.987100    2562 scope.go:117] "RemoveContainer" containerID="2e03fa6ab734d873bd3d6817a799d0c6bdc4591baaa967e2f9a78fe0e36efc47"
Jul 23 18:47:55.987251 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:55.987136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583912112"
Jul 23 18:47:56.995240 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:56.995210    2562 generic.go:356] "Generic (PLEG): container finished" podID="9212648b-6565-477e-8b36-fab63074c683" containerID="b6bc22f43260d5178437a4dd7d71f296f428064c471e4e78708d2abebe5a3665" exitCode=0
Jul 23 18:47:56.995739 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:56.995288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583911856" event={"ID":"9212648b-6565-477e-8b36-fab63074c683","Type":"ContainerDied","Data":"b6bc22f43260d5178437a4dd7d71f296f428064c471e4e78708d2abebe5a3665"}
Jul 23 18:47:58.136143 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.136114    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:58.156264 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.156235    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7426-140622583911856_9212648b-6565-477e-8b36-fab63074c683/test-curl-7426-140622583911856/0.log"
Jul 23 18:47:58.185985 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.185936    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-140622583911856"]
Jul 23 18:47:58.190330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.190302    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-140622583911856"]
Jul 23 18:47:58.197252 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.197222    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sshqs\" (UniqueName: \"kubernetes.io/projected/9212648b-6565-477e-8b36-fab63074c683-kube-api-access-sshqs\") pod \"9212648b-6565-477e-8b36-fab63074c683\" (UID: \"9212648b-6565-477e-8b36-fab63074c683\") "
Jul 23 18:47:58.199666 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.199630    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9212648b-6565-477e-8b36-fab63074c683-kube-api-access-sshqs" (OuterVolumeSpecName: "kube-api-access-sshqs") pod "9212648b-6565-477e-8b36-fab63074c683" (UID: "9212648b-6565-477e-8b36-fab63074c683"). InnerVolumeSpecName "kube-api-access-sshqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:47:58.298713 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.298668    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sshqs\" (UniqueName: \"kubernetes.io/projected/9212648b-6565-477e-8b36-fab63074c683-kube-api-access-sshqs\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:47:58.358756 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.358716    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:47:58.359141 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.359123    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9212648b-6565-477e-8b36-fab63074c683" containerName="test-curl-7426-140622583911856"
Jul 23 18:47:58.359141 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.359138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9212648b-6565-477e-8b36-fab63074c683" containerName="test-curl-7426-140622583911856"
Jul 23 18:47:58.359242 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.359203    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9212648b-6565-477e-8b36-fab63074c683" containerName="test-curl-7426-140622583911856"
Jul 23 18:47:58.362142 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.362126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:47:58.369631 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.369487    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:47:58.399746 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.399704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmb7h\" (UniqueName: \"kubernetes.io/projected/eaba22a5-17c5-4283-a409-8ae48330b5cd-kube-api-access-dmb7h\") pod \"test-curl-7426-140622584028736\" (UID: \"eaba22a5-17c5-4283-a409-8ae48330b5cd\") " pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:47:58.500487 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.500441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmb7h\" (UniqueName: \"kubernetes.io/projected/eaba22a5-17c5-4283-a409-8ae48330b5cd-kube-api-access-dmb7h\") pod \"test-curl-7426-140622584028736\" (UID: \"eaba22a5-17c5-4283-a409-8ae48330b5cd\") " pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:47:58.509727 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.509686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmb7h\" (UniqueName: \"kubernetes.io/projected/eaba22a5-17c5-4283-a409-8ae48330b5cd-kube-api-access-dmb7h\") pod \"test-curl-7426-140622584028736\" (UID: \"eaba22a5-17c5-4283-a409-8ae48330b5cd\") " pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:47:58.672714 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.672631    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:47:58.810094 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:58.810052    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:47:58.812906 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:47:58.812879    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaba22a5_17c5_4283_a409_8ae48330b5cd.slice/crio-099379a3e0f0847f976b8f949a506f8a044cf4feb36befd933efc90c30da8ecb WatchSource:0}: Error finding container 099379a3e0f0847f976b8f949a506f8a044cf4feb36befd933efc90c30da8ecb: Status 404 returned error can't find the container with id 099379a3e0f0847f976b8f949a506f8a044cf4feb36befd933efc90c30da8ecb
Jul 23 18:47:59.007919 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:59.007880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622584028736" event={"ID":"eaba22a5-17c5-4283-a409-8ae48330b5cd","Type":"ContainerStarted","Data":"099379a3e0f0847f976b8f949a506f8a044cf4feb36befd933efc90c30da8ecb"}
Jul 23 18:47:59.009220 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:59.009201    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:59.009333 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:59.009239    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5517712bc180c804073a626ec83f6d254745a203ea30223e2f6ffceeb4e829b4"
Jul 23 18:47:59.020585 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:47:59.020552    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7426-140622583911856\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="9212648b-6565-477e-8b36-fab63074c683" pod="opendatahub/test-curl-7426-140622583911856"
Jul 23 18:47:59.608509 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:47:59.608474    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9212648b-6565-477e-8b36-fab63074c683" path="/var/lib/kubelet/pods/9212648b-6565-477e-8b36-fab63074c683/volumes"
Jul 23 18:48:00.017672 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:00.017633    2562 generic.go:356] "Generic (PLEG): container finished" podID="eaba22a5-17c5-4283-a409-8ae48330b5cd" containerID="debe41b289e8ddd0c723719a54b5bece55a408dec587b37ef8a788ea013de4ce" exitCode=0
Jul 23 18:48:00.017886 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:00.017690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622584028736" event={"ID":"eaba22a5-17c5-4283-a409-8ae48330b5cd","Type":"ContainerDied","Data":"debe41b289e8ddd0c723719a54b5bece55a408dec587b37ef8a788ea013de4ce"}
Jul 23 18:48:01.166229 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.166202    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:01.204899 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.204873    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7426-140622584028736_eaba22a5-17c5-4283-a409-8ae48330b5cd/test-curl-7426-140622584028736/0.log"
Jul 23 18:48:01.226279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.226242    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmb7h\" (UniqueName: \"kubernetes.io/projected/eaba22a5-17c5-4283-a409-8ae48330b5cd-kube-api-access-dmb7h\") pod \"eaba22a5-17c5-4283-a409-8ae48330b5cd\" (UID: \"eaba22a5-17c5-4283-a409-8ae48330b5cd\") "
Jul 23 18:48:01.228516 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.228482    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaba22a5-17c5-4283-a409-8ae48330b5cd-kube-api-access-dmb7h" (OuterVolumeSpecName: "kube-api-access-dmb7h") pod "eaba22a5-17c5-4283-a409-8ae48330b5cd" (UID: "eaba22a5-17c5-4283-a409-8ae48330b5cd"). InnerVolumeSpecName "kube-api-access-dmb7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:01.248793 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.248762    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:01.259129 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.259097    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:01.327657 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.327561    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmb7h\" (UniqueName: \"kubernetes.io/projected/eaba22a5-17c5-4283-a409-8ae48330b5cd-kube-api-access-dmb7h\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:01.391042 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.391008    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:01.391460 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.391445    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaba22a5-17c5-4283-a409-8ae48330b5cd" containerName="test-curl-7426-140622584028736"
Jul 23 18:48:01.391517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.391463    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaba22a5-17c5-4283-a409-8ae48330b5cd" containerName="test-curl-7426-140622584028736"
Jul 23 18:48:01.391558 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.391535    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaba22a5-17c5-4283-a409-8ae48330b5cd" containerName="test-curl-7426-140622584028736"
Jul 23 18:48:01.394814 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.394795    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:01.403852 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.403818    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:01.529556 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.529520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qql\" (UniqueName: \"kubernetes.io/projected/38226468-c7ba-4aae-b378-07a1e18e9dbe-kube-api-access-d5qql\") pod \"test-curl-7426-140622584028736\" (UID: \"38226468-c7ba-4aae-b378-07a1e18e9dbe\") " pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:01.610040 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.609958    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaba22a5-17c5-4283-a409-8ae48330b5cd" path="/var/lib/kubelet/pods/eaba22a5-17c5-4283-a409-8ae48330b5cd/volumes"
Jul 23 18:48:01.630171 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.630142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qql\" (UniqueName: \"kubernetes.io/projected/38226468-c7ba-4aae-b378-07a1e18e9dbe-kube-api-access-d5qql\") pod \"test-curl-7426-140622584028736\" (UID: \"38226468-c7ba-4aae-b378-07a1e18e9dbe\") " pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:01.639307 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.639282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qql\" (UniqueName: \"kubernetes.io/projected/38226468-c7ba-4aae-b378-07a1e18e9dbe-kube-api-access-d5qql\") pod \"test-curl-7426-140622584028736\" (UID: \"38226468-c7ba-4aae-b378-07a1e18e9dbe\") " pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:01.706988 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.706927    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:01.845383 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:01.845350    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:01.847993 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:48:01.847935    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38226468_c7ba_4aae_b378_07a1e18e9dbe.slice/crio-073d12754ce130c5ab079f4a92b55cfc8fbc62ee7764af4312b52e6e89785896 WatchSource:0}: Error finding container 073d12754ce130c5ab079f4a92b55cfc8fbc62ee7764af4312b52e6e89785896: Status 404 returned error can't find the container with id 073d12754ce130c5ab079f4a92b55cfc8fbc62ee7764af4312b52e6e89785896
Jul 23 18:48:02.033762 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:02.033728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622584028736" event={"ID":"38226468-c7ba-4aae-b378-07a1e18e9dbe","Type":"ContainerStarted","Data":"073d12754ce130c5ab079f4a92b55cfc8fbc62ee7764af4312b52e6e89785896"}
Jul 23 18:48:02.035030 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:02.035003    2562 scope.go:117] "RemoveContainer" containerID="debe41b289e8ddd0c723719a54b5bece55a408dec587b37ef8a788ea013de4ce"
Jul 23 18:48:02.035162 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:02.035044    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:03.044054 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:03.044017    2562 generic.go:356] "Generic (PLEG): container finished" podID="38226468-c7ba-4aae-b378-07a1e18e9dbe" containerID="86aa72927f2d4c8009e44252da929ff911492026feb3016ec13b76575dd6a0be" exitCode=0
Jul 23 18:48:03.044447 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:03.044077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622584028736" event={"ID":"38226468-c7ba-4aae-b378-07a1e18e9dbe","Type":"ContainerDied","Data":"86aa72927f2d4c8009e44252da929ff911492026feb3016ec13b76575dd6a0be"}
Jul 23 18:48:04.192053 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.192026    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:04.212511 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.212478    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7426-140622584028736_38226468-c7ba-4aae-b378-07a1e18e9dbe/test-curl-7426-140622584028736/0.log"
Jul 23 18:48:04.240313 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.240278    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:04.245693 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.245661    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-140622584028736"]
Jul 23 18:48:04.251812 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.251788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qql\" (UniqueName: \"kubernetes.io/projected/38226468-c7ba-4aae-b378-07a1e18e9dbe-kube-api-access-d5qql\") pod \"38226468-c7ba-4aae-b378-07a1e18e9dbe\" (UID: \"38226468-c7ba-4aae-b378-07a1e18e9dbe\") "
Jul 23 18:48:04.254121 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.254092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38226468-c7ba-4aae-b378-07a1e18e9dbe-kube-api-access-d5qql" (OuterVolumeSpecName: "kube-api-access-d5qql") pod "38226468-c7ba-4aae-b378-07a1e18e9dbe" (UID: "38226468-c7ba-4aae-b378-07a1e18e9dbe"). InnerVolumeSpecName "kube-api-access-d5qql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:04.353310 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.353218    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qql\" (UniqueName: \"kubernetes.io/projected/38226468-c7ba-4aae-b378-07a1e18e9dbe-kube-api-access-d5qql\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:04.387014 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.386980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:04.387390 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.387377    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38226468-c7ba-4aae-b378-07a1e18e9dbe" containerName="test-curl-7426-140622584028736"
Jul 23 18:48:04.387443 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.387391    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="38226468-c7ba-4aae-b378-07a1e18e9dbe" containerName="test-curl-7426-140622584028736"
Jul 23 18:48:04.387475 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.387456    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="38226468-c7ba-4aae-b378-07a1e18e9dbe" containerName="test-curl-7426-140622584028736"
Jul 23 18:48:04.390422 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.390406    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:04.400634 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.400212    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:04.454549 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.454504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcthr\" (UniqueName: \"kubernetes.io/projected/55b372d6-b742-49a8-b4de-a9af7b569fa4-kube-api-access-kcthr\") pod \"test-curl-7426-140622583985616\" (UID: \"55b372d6-b742-49a8-b4de-a9af7b569fa4\") " pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:04.555853 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.555801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcthr\" (UniqueName: \"kubernetes.io/projected/55b372d6-b742-49a8-b4de-a9af7b569fa4-kube-api-access-kcthr\") pod \"test-curl-7426-140622583985616\" (UID: \"55b372d6-b742-49a8-b4de-a9af7b569fa4\") " pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:04.569743 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.569708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcthr\" (UniqueName: \"kubernetes.io/projected/55b372d6-b742-49a8-b4de-a9af7b569fa4-kube-api-access-kcthr\") pod \"test-curl-7426-140622583985616\" (UID: \"55b372d6-b742-49a8-b4de-a9af7b569fa4\") " pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:04.702801 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.702712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:04.850426 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:04.850387    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:04.851355 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:48:04.851331    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b372d6_b742_49a8_b4de_a9af7b569fa4.slice/crio-07782b08714b011de8ddf4721a625553101c387600110321deb8c1496633d52a WatchSource:0}: Error finding container 07782b08714b011de8ddf4721a625553101c387600110321deb8c1496633d52a: Status 404 returned error can't find the container with id 07782b08714b011de8ddf4721a625553101c387600110321deb8c1496633d52a
Jul 23 18:48:05.062817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:05.062785    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="073d12754ce130c5ab079f4a92b55cfc8fbc62ee7764af4312b52e6e89785896"
Jul 23 18:48:05.062817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:05.062794    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:05.064230 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:05.064199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583985616" event={"ID":"55b372d6-b742-49a8-b4de-a9af7b569fa4","Type":"ContainerStarted","Data":"07782b08714b011de8ddf4721a625553101c387600110321deb8c1496633d52a"}
Jul 23 18:48:05.072927 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:48:05.072898    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7426-140622584028736\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="38226468-c7ba-4aae-b378-07a1e18e9dbe" pod="opendatahub/test-curl-7426-140622584028736"
Jul 23 18:48:05.608666 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:05.608629    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38226468-c7ba-4aae-b378-07a1e18e9dbe" path="/var/lib/kubelet/pods/38226468-c7ba-4aae-b378-07a1e18e9dbe/volumes"
Jul 23 18:48:06.070862 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:06.070829    2562 generic.go:356] "Generic (PLEG): container finished" podID="55b372d6-b742-49a8-b4de-a9af7b569fa4" containerID="29b0eab8734657b9fc533e60e25c549e97115f89293ae8819c83f1ba65d30cf0" exitCode=0
Jul 23 18:48:06.070862 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:06.070867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583985616" event={"ID":"55b372d6-b742-49a8-b4de-a9af7b569fa4","Type":"ContainerDied","Data":"29b0eab8734657b9fc533e60e25c549e97115f89293ae8819c83f1ba65d30cf0"}
Jul 23 18:48:07.232137 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.232104    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:07.252170 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.252140    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7426-140622583985616_55b372d6-b742-49a8-b4de-a9af7b569fa4/test-curl-7426-140622583985616/0.log"
Jul 23 18:48:07.274604 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.274568    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:07.278408 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.278378    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:07.295774 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.295747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcthr\" (UniqueName: \"kubernetes.io/projected/55b372d6-b742-49a8-b4de-a9af7b569fa4-kube-api-access-kcthr\") pod \"55b372d6-b742-49a8-b4de-a9af7b569fa4\" (UID: \"55b372d6-b742-49a8-b4de-a9af7b569fa4\") "
Jul 23 18:48:07.297929 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.297903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b372d6-b742-49a8-b4de-a9af7b569fa4-kube-api-access-kcthr" (OuterVolumeSpecName: "kube-api-access-kcthr") pod "55b372d6-b742-49a8-b4de-a9af7b569fa4" (UID: "55b372d6-b742-49a8-b4de-a9af7b569fa4"). InnerVolumeSpecName "kube-api-access-kcthr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:07.396746 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.396643    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcthr\" (UniqueName: \"kubernetes.io/projected/55b372d6-b742-49a8-b4de-a9af7b569fa4-kube-api-access-kcthr\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:07.409107 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.409074    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:07.409498 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.409484    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b372d6-b742-49a8-b4de-a9af7b569fa4" containerName="test-curl-7426-140622583985616"
Jul 23 18:48:07.409536 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.409500    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b372d6-b742-49a8-b4de-a9af7b569fa4" containerName="test-curl-7426-140622583985616"
Jul 23 18:48:07.409580 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.409571    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b372d6-b742-49a8-b4de-a9af7b569fa4" containerName="test-curl-7426-140622583985616"
Jul 23 18:48:07.412682 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.412665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:07.416637 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.416606    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:07.498121 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.498084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6lz\" (UniqueName: \"kubernetes.io/projected/00aef122-cb84-4d2b-88f6-4823b85f1205-kube-api-access-zb6lz\") pod \"test-curl-7426-140622583985616\" (UID: \"00aef122-cb84-4d2b-88f6-4823b85f1205\") " pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:07.598618 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.598583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6lz\" (UniqueName: \"kubernetes.io/projected/00aef122-cb84-4d2b-88f6-4823b85f1205-kube-api-access-zb6lz\") pod \"test-curl-7426-140622583985616\" (UID: \"00aef122-cb84-4d2b-88f6-4823b85f1205\") " pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:07.606495 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.606462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6lz\" (UniqueName: \"kubernetes.io/projected/00aef122-cb84-4d2b-88f6-4823b85f1205-kube-api-access-zb6lz\") pod \"test-curl-7426-140622583985616\" (UID: \"00aef122-cb84-4d2b-88f6-4823b85f1205\") " pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:07.609506 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.609467    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b372d6-b742-49a8-b4de-a9af7b569fa4" path="/var/lib/kubelet/pods/55b372d6-b742-49a8-b4de-a9af7b569fa4/volumes"
Jul 23 18:48:07.723644 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.723554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:07.863002 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:07.862977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:07.865264 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:48:07.865240    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00aef122_cb84_4d2b_88f6_4823b85f1205.slice/crio-69299fe5f375cbdc3fc57e3e6592b33bcf924e4e6326a7ae861e358fc5495a33 WatchSource:0}: Error finding container 69299fe5f375cbdc3fc57e3e6592b33bcf924e4e6326a7ae861e358fc5495a33: Status 404 returned error can't find the container with id 69299fe5f375cbdc3fc57e3e6592b33bcf924e4e6326a7ae861e358fc5495a33
Jul 23 18:48:08.082807 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:08.082769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583985616" event={"ID":"00aef122-cb84-4d2b-88f6-4823b85f1205","Type":"ContainerStarted","Data":"69299fe5f375cbdc3fc57e3e6592b33bcf924e4e6326a7ae861e358fc5495a33"}
Jul 23 18:48:08.084206 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:08.084178    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:08.084342 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:08.084182    2562 scope.go:117] "RemoveContainer" containerID="29b0eab8734657b9fc533e60e25c549e97115f89293ae8819c83f1ba65d30cf0"
Jul 23 18:48:09.093552 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:09.093517    2562 generic.go:356] "Generic (PLEG): container finished" podID="00aef122-cb84-4d2b-88f6-4823b85f1205" containerID="d74029a21ff57156943d16993fd09a84ece9b7575b819e091b4eb082de76ea2f" exitCode=0
Jul 23 18:48:09.094030 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:09.093568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-140622583985616" event={"ID":"00aef122-cb84-4d2b-88f6-4823b85f1205","Type":"ContainerDied","Data":"d74029a21ff57156943d16993fd09a84ece9b7575b819e091b4eb082de76ea2f"}
Jul 23 18:48:10.245390 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.245364    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:10.265776 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.265744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7426-140622583985616_00aef122-cb84-4d2b-88f6-4823b85f1205/test-curl-7426-140622583985616/0.log"
Jul 23 18:48:10.289206 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.289173    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:10.293517 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.293488    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-140622583985616"]
Jul 23 18:48:10.323222 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.323193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb6lz\" (UniqueName: \"kubernetes.io/projected/00aef122-cb84-4d2b-88f6-4823b85f1205-kube-api-access-zb6lz\") pod \"00aef122-cb84-4d2b-88f6-4823b85f1205\" (UID: \"00aef122-cb84-4d2b-88f6-4823b85f1205\") "
Jul 23 18:48:10.325503 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.325479    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00aef122-cb84-4d2b-88f6-4823b85f1205-kube-api-access-zb6lz" (OuterVolumeSpecName: "kube-api-access-zb6lz") pod "00aef122-cb84-4d2b-88f6-4823b85f1205" (UID: "00aef122-cb84-4d2b-88f6-4823b85f1205"). InnerVolumeSpecName "kube-api-access-zb6lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:10.424323 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.424226    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6lz\" (UniqueName: \"kubernetes.io/projected/00aef122-cb84-4d2b-88f6-4823b85f1205-kube-api-access-zb6lz\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:10.502622 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.502589    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"]
Jul 23 18:48:10.503257 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.503240    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00aef122-cb84-4d2b-88f6-4823b85f1205" containerName="test-curl-7426-140622583985616"
Jul 23 18:48:10.503308 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.503262    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00aef122-cb84-4d2b-88f6-4823b85f1205" containerName="test-curl-7426-140622583985616"
Jul 23 18:48:10.503381 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.503369    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00aef122-cb84-4d2b-88f6-4823b85f1205" containerName="test-curl-7426-140622583985616"
Jul 23 18:48:10.506876 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.506854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:10.509359 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.509336    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:48:10.514619 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.514594    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"]
Jul 23 18:48:10.625812 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.625776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c\" (UID: \"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:10.727135 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.727041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c\" (UID: \"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:10.727811 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.727789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c\" (UID: \"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:10.821117 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.821079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:10.960306 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:10.960276    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"]
Jul 23 18:48:10.962243 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:48:10.962219    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76b2b50_ffa2_49ca_bba0_328f8fbea4f8.slice/crio-8d7440b59c80b4c0a3078a8c4c4877ea797be5b7a706c8dac11ba6faa04dfed6 WatchSource:0}: Error finding container 8d7440b59c80b4c0a3078a8c4c4877ea797be5b7a706c8dac11ba6faa04dfed6: Status 404 returned error can't find the container with id 8d7440b59c80b4c0a3078a8c4c4877ea797be5b7a706c8dac11ba6faa04dfed6
Jul 23 18:48:11.107019 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:11.106988    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:11.107019 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:11.107004    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69299fe5f375cbdc3fc57e3e6592b33bcf924e4e6326a7ae861e358fc5495a33"
Jul 23 18:48:11.108380 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:11.108351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c" event={"ID":"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8","Type":"ContainerStarted","Data":"8d7440b59c80b4c0a3078a8c4c4877ea797be5b7a706c8dac11ba6faa04dfed6"}
Jul 23 18:48:11.116749 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:48:11.116724    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7426-140622583985616\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="00aef122-cb84-4d2b-88f6-4823b85f1205" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:11.606764 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:48:11.606676    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7426-140622583985616\" is forbidden: User \"system:node:ip-10-0-130-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-3.ec2.internal' and this object" podUID="00aef122-cb84-4d2b-88f6-4823b85f1205" pod="opendatahub/test-curl-7426-140622583985616"
Jul 23 18:48:11.608842 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:11.608818    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00aef122-cb84-4d2b-88f6-4823b85f1205" path="/var/lib/kubelet/pods/00aef122-cb84-4d2b-88f6-4823b85f1205/volumes"
Jul 23 18:48:12.115861 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:12.115823    2562 generic.go:356] "Generic (PLEG): container finished" podID="b76b2b50-ffa2-49ca-bba0-328f8fbea4f8" containerID="a6445c96303d8e4438330a3f28f501029781f5311d4fdf0c2c0110ec0a0787fd" exitCode=0
Jul 23 18:48:12.116076 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:12.115885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c" event={"ID":"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8","Type":"ContainerDied","Data":"a6445c96303d8e4438330a3f28f501029781f5311d4fdf0c2c0110ec0a0787fd"}
Jul 23 18:48:13.262401 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:13.262372    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:13.353839 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:13.353801    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8-maas-api-service-ca\") pod \"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8\" (UID: \"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8\") "
Jul 23 18:48:13.354247 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:13.354226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b76b2b50-ffa2-49ca-bba0-328f8fbea4f8" (UID: "b76b2b50-ffa2-49ca-bba0-328f8fbea4f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:48:13.454618 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:13.454534    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:14.128715 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:14.128675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c" event={"ID":"b76b2b50-ffa2-49ca-bba0-328f8fbea4f8","Type":"ContainerDied","Data":"8d7440b59c80b4c0a3078a8c4c4877ea797be5b7a706c8dac11ba6faa04dfed6"}
Jul 23 18:48:14.128865 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:14.128730    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d7440b59c80b4c0a3078a8c4c4877ea797be5b7a706c8dac11ba6faa04dfed6"
Jul 23 18:48:14.128865 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:14.128693    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"
Jul 23 18:48:20.758540 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:20.758501    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"]
Jul 23 18:48:20.758926 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:20.758742    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj" podUID="1eb012b0-432e-429b-a0f1-ce65e32350ff" containerName="maas-api" containerID="cri-o://c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7" gracePeriod=30
Jul 23 18:48:20.783865 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:20.783834    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"]
Jul 23 18:48:20.788360 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:20.788335    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5f06b1-29747205-jsvwz"]
Jul 23 18:48:21.018183 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.018119    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:48:21.124753 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.124720    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb012b0-432e-429b-a0f1-ce65e32350ff-openshift-service-ca\") pod \"1eb012b0-432e-429b-a0f1-ce65e32350ff\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") "
Jul 23 18:48:21.124924 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.124776    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn6hn\" (UniqueName: \"kubernetes.io/projected/1eb012b0-432e-429b-a0f1-ce65e32350ff-kube-api-access-dn6hn\") pod \"1eb012b0-432e-429b-a0f1-ce65e32350ff\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") "
Jul 23 18:48:21.124924 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.124835    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb012b0-432e-429b-a0f1-ce65e32350ff-maas-api-tls\") pod \"1eb012b0-432e-429b-a0f1-ce65e32350ff\" (UID: \"1eb012b0-432e-429b-a0f1-ce65e32350ff\") "
Jul 23 18:48:21.125192 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.125160    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb012b0-432e-429b-a0f1-ce65e32350ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1eb012b0-432e-429b-a0f1-ce65e32350ff" (UID: "1eb012b0-432e-429b-a0f1-ce65e32350ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:48:21.127135 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.127112    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb012b0-432e-429b-a0f1-ce65e32350ff-kube-api-access-dn6hn" (OuterVolumeSpecName: "kube-api-access-dn6hn") pod "1eb012b0-432e-429b-a0f1-ce65e32350ff" (UID: "1eb012b0-432e-429b-a0f1-ce65e32350ff"). InnerVolumeSpecName "kube-api-access-dn6hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:21.127211 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.127131    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb012b0-432e-429b-a0f1-ce65e32350ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1eb012b0-432e-429b-a0f1-ce65e32350ff" (UID: "1eb012b0-432e-429b-a0f1-ce65e32350ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:48:21.170559 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.170524    2562 generic.go:356] "Generic (PLEG): container finished" podID="1eb012b0-432e-429b-a0f1-ce65e32350ff" containerID="c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7" exitCode=0
Jul 23 18:48:21.170699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.170580    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"
Jul 23 18:48:21.170699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.170616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj" event={"ID":"1eb012b0-432e-429b-a0f1-ce65e32350ff","Type":"ContainerDied","Data":"c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7"}
Jul 23 18:48:21.170699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.170666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj" event={"ID":"1eb012b0-432e-429b-a0f1-ce65e32350ff","Type":"ContainerDied","Data":"76c766473865f0101e759e08d6826a4b0fb03b96efa32a61a7d24ef6f7333164"}
Jul 23 18:48:21.170699 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.170687    2562 scope.go:117] "RemoveContainer" containerID="c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7"
Jul 23 18:48:21.180800 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.180784    2562 scope.go:117] "RemoveContainer" containerID="c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7"
Jul 23 18:48:21.181068 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:48:21.181050    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7\": container with ID starting with c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7 not found: ID does not exist" containerID="c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7"
Jul 23 18:48:21.181132 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.181078    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7"} err="failed to get container status \"c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7\": rpc error: code = NotFound desc = could not find container \"c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7\": container with ID starting with c6cb3df433ce9f7663897518f6cc8df2dab9f74a60ebb09de8b8871781a994e7 not found: ID does not exist"
Jul 23 18:48:21.200713 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.200680    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"]
Jul 23 18:48:21.206823 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.206792    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f06b1-5479bffb9-8ltqj"]
Jul 23 18:48:21.225438 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.225411    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb012b0-432e-429b-a0f1-ce65e32350ff-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:21.225438 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.225435    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6hn\" (UniqueName: \"kubernetes.io/projected/1eb012b0-432e-429b-a0f1-ce65e32350ff-kube-api-access-dn6hn\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:21.225582 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.225447    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb012b0-432e-429b-a0f1-ce65e32350ff-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:21.609789 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.609754    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb012b0-432e-429b-a0f1-ce65e32350ff" path="/var/lib/kubelet/pods/1eb012b0-432e-429b-a0f1-ce65e32350ff/volumes"
Jul 23 18:48:21.610087 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:21.610072    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f05ff318-b364-4d32-a039-0209dfd5732f" path="/var/lib/kubelet/pods/f05ff318-b364-4d32-a039-0209dfd5732f/volumes"
Jul 23 18:48:25.657764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:25.657730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"]
Jul 23 18:48:25.662606 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:25.662576    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5f06b1-q2s6c"]
Jul 23 18:48:27.612586 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:27.612551    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76b2b50-ffa2-49ca-bba0-328f8fbea4f8" path="/var/lib/kubelet/pods/b76b2b50-ffa2-49ca-bba0-328f8fbea4f8/volumes"
Jul 23 18:48:33.230474 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230433    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"]
Jul 23 18:48:33.230866 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230828    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76b2b50-ffa2-49ca-bba0-328f8fbea4f8" containerName="revoke-keys"
Jul 23 18:48:33.230866 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230840    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76b2b50-ffa2-49ca-bba0-328f8fbea4f8" containerName="revoke-keys"
Jul 23 18:48:33.230866 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230850    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb012b0-432e-429b-a0f1-ce65e32350ff" containerName="maas-api"
Jul 23 18:48:33.230866 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230856    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb012b0-432e-429b-a0f1-ce65e32350ff" containerName="maas-api"
Jul 23 18:48:33.231034 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230927    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76b2b50-ffa2-49ca-bba0-328f8fbea4f8" containerName="revoke-keys"
Jul 23 18:48:33.231034 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.230936    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb012b0-432e-429b-a0f1-ce65e32350ff" containerName="maas-api"
Jul 23 18:48:33.234026 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.234008    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:33.236896 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.236869    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmqxt\""
Jul 23 18:48:33.243471 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.243447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"]
Jul 23 18:48:33.324537 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.324504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/431e6970-c31b-4e90-b224-e389defa9532-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l\" (UID: \"431e6970-c31b-4e90-b224-e389defa9532\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:33.425274 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.425238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/431e6970-c31b-4e90-b224-e389defa9532-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l\" (UID: \"431e6970-c31b-4e90-b224-e389defa9532\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:33.425862 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.425838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/431e6970-c31b-4e90-b224-e389defa9532-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l\" (UID: \"431e6970-c31b-4e90-b224-e389defa9532\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:33.545035 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.544999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:33.686068 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:33.686045    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"]
Jul 23 18:48:33.687635 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:48:33.687603    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431e6970_c31b_4e90_b224_e389defa9532.slice/crio-e565c983a4389527ac63d948b7a929e058a792f917740b7212c4a553a2ed7bc2 WatchSource:0}: Error finding container e565c983a4389527ac63d948b7a929e058a792f917740b7212c4a553a2ed7bc2: Status 404 returned error can't find the container with id e565c983a4389527ac63d948b7a929e058a792f917740b7212c4a553a2ed7bc2
Jul 23 18:48:34.247080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:34.246979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l" event={"ID":"431e6970-c31b-4e90-b224-e389defa9532","Type":"ContainerStarted","Data":"4c48c388ed131b1f10210a6b0b5597f3828b7499c8dd8af27ed451f3aba6098e"}
Jul 23 18:48:34.247080 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:34.247040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l" event={"ID":"431e6970-c31b-4e90-b224-e389defa9532","Type":"ContainerStarted","Data":"e565c983a4389527ac63d948b7a929e058a792f917740b7212c4a553a2ed7bc2"}
Jul 23 18:48:34.264637 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:34.264585    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l" podStartSLOduration=1.264567556 podStartE2EDuration="1.264567556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:48:34.261879246 +0000 UTC m=+2953.052036363" watchObservedRunningTime="2026-07-23 18:48:34.264567556 +0000 UTC m=+2953.054724670"
Jul 23 18:48:35.254034 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:35.253998    2562 generic.go:356] "Generic (PLEG): container finished" podID="431e6970-c31b-4e90-b224-e389defa9532" containerID="4c48c388ed131b1f10210a6b0b5597f3828b7499c8dd8af27ed451f3aba6098e" exitCode=0
Jul 23 18:48:35.254420 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:35.254095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l" event={"ID":"431e6970-c31b-4e90-b224-e389defa9532","Type":"ContainerDied","Data":"4c48c388ed131b1f10210a6b0b5597f3828b7499c8dd8af27ed451f3aba6098e"}
Jul 23 18:48:36.395846 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:36.395822    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:36.448781 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:36.448744    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/431e6970-c31b-4e90-b224-e389defa9532-maas-api-service-ca\") pod \"431e6970-c31b-4e90-b224-e389defa9532\" (UID: \"431e6970-c31b-4e90-b224-e389defa9532\") "
Jul 23 18:48:36.449115 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:36.449093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431e6970-c31b-4e90-b224-e389defa9532-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "431e6970-c31b-4e90-b224-e389defa9532" (UID: "431e6970-c31b-4e90-b224-e389defa9532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:48:36.549698 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:36.549614    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/431e6970-c31b-4e90-b224-e389defa9532-maas-api-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:37.274041 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:37.273994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l" event={"ID":"431e6970-c31b-4e90-b224-e389defa9532","Type":"ContainerDied","Data":"e565c983a4389527ac63d948b7a929e058a792f917740b7212c4a553a2ed7bc2"}
Jul 23 18:48:37.274214 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:37.274075    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e565c983a4389527ac63d948b7a929e058a792f917740b7212c4a553a2ed7bc2"
Jul 23 18:48:37.274214 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:37.274012    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"
Jul 23 18:48:44.976572 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:44.976477    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"]
Jul 23 18:48:44.977051 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:44.976980    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw" podUID="b87eb14c-921a-45df-8ec7-b9f0f8a04588" containerName="maas-api" containerID="cri-o://7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9" gracePeriod=30
Jul 23 18:48:45.004004 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.003974    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"]
Jul 23 18:48:45.006093 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.006052    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8ee985-29747205-j6vqt"]
Jul 23 18:48:45.238289 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.238234    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:48:45.319932 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.319891    2562 generic.go:356] "Generic (PLEG): container finished" podID="b87eb14c-921a-45df-8ec7-b9f0f8a04588" containerID="7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9" exitCode=0
Jul 23 18:48:45.320120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.319970    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"
Jul 23 18:48:45.320120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.319977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw" event={"ID":"b87eb14c-921a-45df-8ec7-b9f0f8a04588","Type":"ContainerDied","Data":"7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9"}
Jul 23 18:48:45.320120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.320039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw" event={"ID":"b87eb14c-921a-45df-8ec7-b9f0f8a04588","Type":"ContainerDied","Data":"50f622a4158450d1242da99469d2fd752260af5cf6559da8004b837c22035e7f"}
Jul 23 18:48:45.320120 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.320063    2562 scope.go:117] "RemoveContainer" containerID="7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9"
Jul 23 18:48:45.325712 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.325692    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2b22\" (UniqueName: \"kubernetes.io/projected/b87eb14c-921a-45df-8ec7-b9f0f8a04588-kube-api-access-s2b22\") pod \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") "
Jul 23 18:48:45.325815 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.325737    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87eb14c-921a-45df-8ec7-b9f0f8a04588-maas-api-tls\") pod \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") "
Jul 23 18:48:45.325815 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.325764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87eb14c-921a-45df-8ec7-b9f0f8a04588-openshift-service-ca\") pod \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\" (UID: \"b87eb14c-921a-45df-8ec7-b9f0f8a04588\") "
Jul 23 18:48:45.326187 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.326163    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87eb14c-921a-45df-8ec7-b9f0f8a04588-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b87eb14c-921a-45df-8ec7-b9f0f8a04588" (UID: "b87eb14c-921a-45df-8ec7-b9f0f8a04588"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:48:45.327831 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.327804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87eb14c-921a-45df-8ec7-b9f0f8a04588-kube-api-access-s2b22" (OuterVolumeSpecName: "kube-api-access-s2b22") pod "b87eb14c-921a-45df-8ec7-b9f0f8a04588" (UID: "b87eb14c-921a-45df-8ec7-b9f0f8a04588"). InnerVolumeSpecName "kube-api-access-s2b22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:45.328119 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.328093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87eb14c-921a-45df-8ec7-b9f0f8a04588-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b87eb14c-921a-45df-8ec7-b9f0f8a04588" (UID: "b87eb14c-921a-45df-8ec7-b9f0f8a04588"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:48:45.329789 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.329773    2562 scope.go:117] "RemoveContainer" containerID="7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9"
Jul 23 18:48:45.330093 ip-10-0-130-3 kubenswrapper[2562]: E0723 18:48:45.330070    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9\": container with ID starting with 7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9 not found: ID does not exist" containerID="7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9"
Jul 23 18:48:45.330186 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.330100    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9"} err="failed to get container status \"7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9\": rpc error: code = NotFound desc = could not find container \"7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9\": container with ID starting with 7623f44b7cb413569825943e4ae0197a31b658bc15f7c5ad1551f126a1fdebb9 not found: ID does not exist"
Jul 23 18:48:45.426644 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.426611    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2b22\" (UniqueName: \"kubernetes.io/projected/b87eb14c-921a-45df-8ec7-b9f0f8a04588-kube-api-access-s2b22\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:45.426644 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.426640    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87eb14c-921a-45df-8ec7-b9f0f8a04588-maas-api-tls\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:45.426644 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.426649    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87eb14c-921a-45df-8ec7-b9f0f8a04588-openshift-service-ca\") on node \"ip-10-0-130-3.ec2.internal\" DevicePath \"\""
Jul 23 18:48:45.608469 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.608433    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a39f1b63-aa99-4059-9902-b85c21cb24df" path="/var/lib/kubelet/pods/a39f1b63-aa99-4059-9902-b85c21cb24df/volumes"
Jul 23 18:48:45.639175 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.639140    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"]
Jul 23 18:48:45.642997 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:45.642969    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8ee985-5b9f46c5bd-vg7sw"]
Jul 23 18:48:47.608333 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:47.608297    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87eb14c-921a-45df-8ec7-b9f0f8a04588" path="/var/lib/kubelet/pods/b87eb14c-921a-45df-8ec7-b9f0f8a04588/volumes"
Jul 23 18:48:48.334802 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:48.334769    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"]
Jul 23 18:48:48.338448 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:48.338408    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8ee985-7q28l"]
Jul 23 18:48:49.608066 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:48:49.608025    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431e6970-c31b-4e90-b224-e389defa9532" path="/var/lib/kubelet/pods/431e6970-c31b-4e90-b224-e389defa9532/volumes"
Jul 23 18:49:02.283959 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:02.283912    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pc744_cdff1e77-7007-4433-baa9-c4a3276613ad/manager/0.log"
Jul 23 18:49:02.423528 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:02.423486    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p5bpm_48f610a2-b455-44a4-8a4e-229a376e4dcc/manager/0.log"
Jul 23 18:49:02.549916 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:02.549832    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86bcf48f9b-ldsm9_91e7e0bb-57d8-4d84-976e-8f92f2f761db/manager/0.log"
Jul 23 18:49:02.678835 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:02.678808    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gngw7_a79a2916-00f0-47b2-b9a2-60bd57c5b596/server/0.log"
Jul 23 18:49:02.804124 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:02.804044    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jg4bl_837026c8-64fa-4afa-b45c-3bbb0d092487/manager/0.log"
Jul 23 18:49:03.229233 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:03.229156    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-wqmd5_f6619904-ed33-46f9-8c3c-95429e190ea9/copy-manifests/0.log"
Jul 23 18:49:03.236273 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:03.236245    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-wqmd5_f6619904-ed33-46f9-8c3c-95429e190ea9/manager/0.log"
Jul 23 18:49:03.752162 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:03.752133    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747175-pnc6g_78e5af57-2870-4ab5-a140-3f0082ff7cc0/cleanup/0.log"
Jul 23 18:49:03.998908 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:03.998876    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747205-6lzq2_b74225b3-0209-47d6-a784-ac0656e6b09d/cleanup/0.log"
Jul 23 18:49:04.118961 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:04.118913    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ccqzj_a287be2e-5a18-4abf-80af-01aca49df024/postgres/0.log"
Jul 23 18:49:04.977881 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:04.977849    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h_c5d28d85-751e-4db0-a300-dd4a1f6d2ac1/util/0.log"
Jul 23 18:49:04.985048 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:04.985029    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h_c5d28d85-751e-4db0-a300-dd4a1f6d2ac1/pull/0.log"
Jul 23 18:49:04.992011 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:04.991992    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h_c5d28d85-751e-4db0-a300-dd4a1f6d2ac1/extract/0.log"
Jul 23 18:49:05.110612 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.110582    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6_0a995e83-dee5-40be-a698-016d9d77087b/util/0.log"
Jul 23 18:49:05.117121 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.117097    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6_0a995e83-dee5-40be-a698-016d9d77087b/pull/0.log"
Jul 23 18:49:05.123740 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.123714    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6_0a995e83-dee5-40be-a698-016d9d77087b/extract/0.log"
Jul 23 18:49:05.242839 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.242761    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5_ccba60c3-b4bf-40ca-bf09-16cf120e2546/util/0.log"
Jul 23 18:49:05.249169 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.249150    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5_ccba60c3-b4bf-40ca-bf09-16cf120e2546/pull/0.log"
Jul 23 18:49:05.255737 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.255719    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5_ccba60c3-b4bf-40ca-bf09-16cf120e2546/extract/0.log"
Jul 23 18:49:05.376709 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.376681    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj_862e57b5-0b63-405f-bee2-529f0b760fcd/util/0.log"
Jul 23 18:49:05.383330 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.383305    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj_862e57b5-0b63-405f-bee2-529f0b760fcd/pull/0.log"
Jul 23 18:49:05.389625 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.389606    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj_862e57b5-0b63-405f-bee2-529f0b760fcd/extract/0.log"
Jul 23 18:49:05.645889 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.645861    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kh4pr_a65123fc-7d50-4a0c-9fab-f291760dac5b/manager/0.log"
Jul 23 18:49:05.773614 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.773583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s77t9_1133f77b-042a-470c-9942-b913b9451c0f/manager/0.log"
Jul 23 18:49:05.892175 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:05.892145    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6gj97_0cda38c1-7ae8-4ccf-a177-21ce6b322104/kuadrant-console-plugin/0.log"
Jul 23 18:49:06.018518 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:06.018480    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2r2vh_e7133930-cd65-48e5-a808-6d0cb0c09d3d/registry-server/0.log"
Jul 23 18:49:06.154825 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:06.154795    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-brq6n_1853da34-be56-4fbf-bcf5-8c1a0231e6fe/manager/0.log"
Jul 23 18:49:06.414088 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:06.414006    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9mtnf_46616c85-549a-4d9e-a03d-c8cf8bd05b86/manager/0.log"
Jul 23 18:49:07.182485 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:07.182453    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77d9cbf8b4-wmqzj_ffd75b15-abb2-404b-9978-7e1252260b6e/kube-auth-proxy/0.log"
Jul 23 18:49:07.439452 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:07.439373    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rl2d4_f8c4aab0-e19b-4717-98bf-0d0bccfc76e6/payload-pre-processing/0.log"
Jul 23 18:49:07.563527 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:07.563496    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wcvnv_fa981909-b02a-4eb8-aff2-3219f398b491/payload-processing/0.log"
Jul 23 18:49:07.688151 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:07.688120    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7cc9cdfd-b5f4s_aab248f1-1f0e-4550-abfb-ccf8deff11e6/router/0.log"
Jul 23 18:49:08.181150 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:08.181125    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bpn5w_9595162f-0a58-4998-bd90-8aaa1a2856aa/main/0.log"
Jul 23 18:49:08.306595 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:08.306568    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rssn4_b30fabef-45f0-4488-a849-8e0901dab260/main/0.log"
Jul 23 18:49:14.256886 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.256846    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpdkp/must-gather-6clw5"]
Jul 23 18:49:14.257317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.257255    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87eb14c-921a-45df-8ec7-b9f0f8a04588" containerName="maas-api"
Jul 23 18:49:14.257317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.257268    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87eb14c-921a-45df-8ec7-b9f0f8a04588" containerName="maas-api"
Jul 23 18:49:14.257317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.257290    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431e6970-c31b-4e90-b224-e389defa9532" containerName="revoke-keys"
Jul 23 18:49:14.257317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.257296    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="431e6970-c31b-4e90-b224-e389defa9532" containerName="revoke-keys"
Jul 23 18:49:14.257464 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.257350    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87eb14c-921a-45df-8ec7-b9f0f8a04588" containerName="maas-api"
Jul 23 18:49:14.257464 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.257358    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="431e6970-c31b-4e90-b224-e389defa9532" containerName="revoke-keys"
Jul 23 18:49:14.260706 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.260684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.263072 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.263051    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpdkp\"/\"kube-root-ca.crt\""
Jul 23 18:49:14.263707 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.263688    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpdkp\"/\"openshift-service-ca.crt\""
Jul 23 18:49:14.263773 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.263692    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cpdkp\"/\"default-dockercfg-m9lj7\""
Jul 23 18:49:14.269772 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.269753    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpdkp/must-gather-6clw5"]
Jul 23 18:49:14.377073 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.377031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlms\" (UniqueName: \"kubernetes.io/projected/e974236b-2cc7-4d99-8ca5-a785181448cd-kube-api-access-cdlms\") pod \"must-gather-6clw5\" (UID: \"e974236b-2cc7-4d99-8ca5-a785181448cd\") " pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.377252 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.377143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e974236b-2cc7-4d99-8ca5-a785181448cd-must-gather-output\") pod \"must-gather-6clw5\" (UID: \"e974236b-2cc7-4d99-8ca5-a785181448cd\") " pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.478371 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.478337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlms\" (UniqueName: \"kubernetes.io/projected/e974236b-2cc7-4d99-8ca5-a785181448cd-kube-api-access-cdlms\") pod \"must-gather-6clw5\" (UID: \"e974236b-2cc7-4d99-8ca5-a785181448cd\") " pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.478606 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.478464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e974236b-2cc7-4d99-8ca5-a785181448cd-must-gather-output\") pod \"must-gather-6clw5\" (UID: \"e974236b-2cc7-4d99-8ca5-a785181448cd\") " pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.478817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.478796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e974236b-2cc7-4d99-8ca5-a785181448cd-must-gather-output\") pod \"must-gather-6clw5\" (UID: \"e974236b-2cc7-4d99-8ca5-a785181448cd\") " pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.487075 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.487047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlms\" (UniqueName: \"kubernetes.io/projected/e974236b-2cc7-4d99-8ca5-a785181448cd-kube-api-access-cdlms\") pod \"must-gather-6clw5\" (UID: \"e974236b-2cc7-4d99-8ca5-a785181448cd\") " pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.571414 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.571327    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpdkp/must-gather-6clw5"
Jul 23 18:49:14.711224 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:14.711197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpdkp/must-gather-6clw5"]
Jul 23 18:49:14.712647 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:49:14.712619    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode974236b_2cc7_4d99_8ca5_a785181448cd.slice/crio-bba2e47c0843b9d301cae9500f68ee2ec8953d6897627c62644505f063e66929 WatchSource:0}: Error finding container bba2e47c0843b9d301cae9500f68ee2ec8953d6897627c62644505f063e66929: Status 404 returned error can't find the container with id bba2e47c0843b9d301cae9500f68ee2ec8953d6897627c62644505f063e66929
Jul 23 18:49:15.490170 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:15.490136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/must-gather-6clw5" event={"ID":"e974236b-2cc7-4d99-8ca5-a785181448cd","Type":"ContainerStarted","Data":"bba2e47c0843b9d301cae9500f68ee2ec8953d6897627c62644505f063e66929"}
Jul 23 18:49:16.501291 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:16.501191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/must-gather-6clw5" event={"ID":"e974236b-2cc7-4d99-8ca5-a785181448cd","Type":"ContainerStarted","Data":"f209c8c27b6824c1a2799d2cb2f1037a16509bb6eb3050b37c1d9c845a4b1a07"}
Jul 23 18:49:16.501291 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:16.501238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/must-gather-6clw5" event={"ID":"e974236b-2cc7-4d99-8ca5-a785181448cd","Type":"ContainerStarted","Data":"1d4000b6655a7954ae40018d0ec74756451798fdf9211d38082d89bb690911f0"}
Jul 23 18:49:16.519554 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:16.519482    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cpdkp/must-gather-6clw5" podStartSLOduration=1.128503938 podStartE2EDuration="2.519458012s" totalImagesPullingTime="1.390954074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:49:16.51906756 +0000 UTC m=+2995.309224676" watchObservedRunningTime="2026-07-23 18:49:16.519458012 +0000 UTC m=+2995.309615128"
Jul 23 18:49:17.733074 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:17.733021    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gdqpw_fcbf1c01-1735-4764-9986-a750528b484e/global-pull-secret-syncer/0.log"
Jul 23 18:49:17.925540 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:17.925494    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c789p_a9f3a7bc-def3-4b79-b099-abf23b7f57a5/konnectivity-agent/0.log"
Jul 23 18:49:18.053006 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:18.052969    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-3.ec2.internal_baee3903119d9c9ebdaea817602fe47e/haproxy/0.log"
Jul 23 18:49:21.882718 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:21.882589    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h_c5d28d85-751e-4db0-a300-dd4a1f6d2ac1/extract/0.log"
Jul 23 18:49:21.916966 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:21.916330    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:49:21.927361 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:21.918742    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h_c5d28d85-751e-4db0-a300-dd4a1f6d2ac1/util/0.log"
Jul 23 18:49:21.937965 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:21.936920    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbfsq_5bfa618f-330c-4aa6-9b3a-9aac777c1cf1/ovn-acl-logging/0.log"
Jul 23 18:49:21.949965 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:21.949326    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lxc4h_c5d28d85-751e-4db0-a300-dd4a1f6d2ac1/pull/0.log"
Jul 23 18:49:21.982540 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:21.981404    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6_0a995e83-dee5-40be-a698-016d9d77087b/extract/0.log"
Jul 23 18:49:22.010857 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.010822    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6_0a995e83-dee5-40be-a698-016d9d77087b/util/0.log"
Jul 23 18:49:22.043260 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.043229    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dzml6_0a995e83-dee5-40be-a698-016d9d77087b/pull/0.log"
Jul 23 18:49:22.097604 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.097563    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5_ccba60c3-b4bf-40ca-bf09-16cf120e2546/extract/0.log"
Jul 23 18:49:22.138846 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.138756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5_ccba60c3-b4bf-40ca-bf09-16cf120e2546/util/0.log"
Jul 23 18:49:22.176288 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.176222    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73475l5_ccba60c3-b4bf-40ca-bf09-16cf120e2546/pull/0.log"
Jul 23 18:49:22.210665 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.210631    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj_862e57b5-0b63-405f-bee2-529f0b760fcd/extract/0.log"
Jul 23 18:49:22.248735 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.248701    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj_862e57b5-0b63-405f-bee2-529f0b760fcd/util/0.log"
Jul 23 18:49:22.273766 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.273728    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2xrj_862e57b5-0b63-405f-bee2-529f0b760fcd/pull/0.log"
Jul 23 18:49:22.981528 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:22.981450    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kh4pr_a65123fc-7d50-4a0c-9fab-f291760dac5b/manager/0.log"
Jul 23 18:49:23.019474 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:23.019417    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s77t9_1133f77b-042a-470c-9942-b913b9451c0f/manager/0.log"
Jul 23 18:49:23.049081 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:23.049051    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6gj97_0cda38c1-7ae8-4ccf-a177-21ce6b322104/kuadrant-console-plugin/0.log"
Jul 23 18:49:23.101456 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:23.101418    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2r2vh_e7133930-cd65-48e5-a808-6d0cb0c09d3d/registry-server/0.log"
Jul 23 18:49:23.275986 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:23.275843    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-brq6n_1853da34-be56-4fbf-bcf5-8c1a0231e6fe/manager/0.log"
Jul 23 18:49:23.496897 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:23.496858    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9mtnf_46616c85-549a-4d9e-a03d-c8cf8bd05b86/manager/0.log"
Jul 23 18:49:25.354529 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:25.354495    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xgn2p_f54a589b-695d-45df-b90d-52d4581118d4/kube-state-metrics/0.log"
Jul 23 18:49:25.396644 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:25.396610    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xgn2p_f54a589b-695d-45df-b90d-52d4581118d4/kube-rbac-proxy-main/0.log"
Jul 23 18:49:25.463272 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:25.463246    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xgn2p_f54a589b-695d-45df-b90d-52d4581118d4/kube-rbac-proxy-self/0.log"
Jul 23 18:49:25.633024 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:25.632926    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mmpd_5ab9927a-f08f-42f7-8668-278ec11cd4d5/node-exporter/0.log"
Jul 23 18:49:25.662443 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:25.662409    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mmpd_5ab9927a-f08f-42f7-8668-278ec11cd4d5/kube-rbac-proxy/0.log"
Jul 23 18:49:25.696023 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:25.695986    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mmpd_5ab9927a-f08f-42f7-8668-278ec11cd4d5/init-textfile/0.log"
Jul 23 18:49:27.458725 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.458690    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"]
Jul 23 18:49:27.465316 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.465284    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.478458 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.478428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"]
Jul 23 18:49:27.521967 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.521843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-lib-modules\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.522167 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.522010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-sys\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.522167 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.522056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-podres\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.522167 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.522138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-proc\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.522317 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.522253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgwx\" (UniqueName: \"kubernetes.io/projected/1e27287e-8895-49f4-9f16-2334e9ed3e72-kube-api-access-rcgwx\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.622909 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.622866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgwx\" (UniqueName: \"kubernetes.io/projected/1e27287e-8895-49f4-9f16-2334e9ed3e72-kube-api-access-rcgwx\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.623126 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.622976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-lib-modules\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.623126 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.623062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-sys\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.623126 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.623085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-podres\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.623279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.623132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-proc\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.623279 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.623240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-proc\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.623681 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.623653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-lib-modules\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.624250 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.624226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-sys\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.624343 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.624318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e27287e-8895-49f4-9f16-2334e9ed3e72-podres\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.632626 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.632595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgwx\" (UniqueName: \"kubernetes.io/projected/1e27287e-8895-49f4-9f16-2334e9ed3e72-kube-api-access-rcgwx\") pod \"perf-node-gather-daemonset-hw75x\" (UID: \"1e27287e-8895-49f4-9f16-2334e9ed3e72\") " pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.779397 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.779360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:27.975453 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:27.975428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"]
Jul 23 18:49:27.977428 ip-10-0-130-3 kubenswrapper[2562]: W0723 18:49:27.977387    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e27287e_8895_49f4_9f16_2334e9ed3e72.slice/crio-a9e7857231e4c3c1532bda2d7aef77552895d58620b549fb24979b9218c0e877 WatchSource:0}: Error finding container a9e7857231e4c3c1532bda2d7aef77552895d58620b549fb24979b9218c0e877: Status 404 returned error can't find the container with id a9e7857231e4c3c1532bda2d7aef77552895d58620b549fb24979b9218c0e877
Jul 23 18:49:28.622746 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:28.622703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x" event={"ID":"1e27287e-8895-49f4-9f16-2334e9ed3e72","Type":"ContainerStarted","Data":"1c6773b4eff4084b72238c1835ab06b246b2ef793a7ad095f3d7adb7bb451dd9"}
Jul 23 18:49:28.623256 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:28.622770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x" event={"ID":"1e27287e-8895-49f4-9f16-2334e9ed3e72","Type":"ContainerStarted","Data":"a9e7857231e4c3c1532bda2d7aef77552895d58620b549fb24979b9218c0e877"}
Jul 23 18:49:28.623631 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:28.623564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:28.642764 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:28.642712    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x" podStartSLOduration=1.6426790439999999 podStartE2EDuration="1.642679044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:49:28.640885363 +0000 UTC m=+3007.431042478" watchObservedRunningTime="2026-07-23 18:49:28.642679044 +0000 UTC m=+3007.432836159"
Jul 23 18:49:29.611345 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:29.611318    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-znpxf_8bb7fffa-642d-4201-ba63-f62a685a566c/volume-data-source-validator/0.log"
Jul 23 18:49:30.716377 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:30.716337    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77vs7_ecda0a51-585e-43b0-89e0-b5c4e0a52004/dns/0.log"
Jul 23 18:49:30.739742 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:30.739712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77vs7_ecda0a51-585e-43b0-89e0-b5c4e0a52004/kube-rbac-proxy/0.log"
Jul 23 18:49:30.867498 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:30.867469    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9xbxt_94942e6d-c9a9-4fb5-bac0-4aa81845e979/dns-node-resolver/0.log"
Jul 23 18:49:31.592547 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:31.592522    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lwdc5_da29655f-7bd7-490f-926b-c40c5353dffc/node-ca/0.log"
Jul 23 18:49:32.853122 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:32.853094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77d9cbf8b4-wmqzj_ffd75b15-abb2-404b-9978-7e1252260b6e/kube-auth-proxy/0.log"
Jul 23 18:49:32.948462 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:32.948428    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rl2d4_f8c4aab0-e19b-4717-98bf-0d0bccfc76e6/payload-pre-processing/0.log"
Jul 23 18:49:32.982342 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:32.982312    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wcvnv_fa981909-b02a-4eb8-aff2-3219f398b491/payload-processing/0.log"
Jul 23 18:49:33.007817 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:33.007791    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7cc9cdfd-b5f4s_aab248f1-1f0e-4550-abfb-ccf8deff11e6/router/0.log"
Jul 23 18:49:33.772578 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:33.772548    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhlzz_cd24080b-11ac-4d1a-85c5-f3928b81d06e/serve-healthcheck-canary/0.log"
Jul 23 18:49:34.263937 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:34.263904    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-f72dk_d1abe68d-f131-43c3-86b5-224632d4bd30/insights-operator/0.log"
Jul 23 18:49:34.265561 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:34.265535    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-f72dk_d1abe68d-f131-43c3-86b5-224632d4bd30/insights-operator/1.log"
Jul 23 18:49:34.440779 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:34.440752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlnq4_9bf9f4de-2e4d-4981-a100-9d1c91ba02f5/kube-rbac-proxy/0.log"
Jul 23 18:49:34.470673 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:34.470644    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlnq4_9bf9f4de-2e4d-4981-a100-9d1c91ba02f5/exporter/0.log"
Jul 23 18:49:34.494127 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:34.494098    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlnq4_9bf9f4de-2e4d-4981-a100-9d1c91ba02f5/extractor/0.log"
Jul 23 18:49:35.655971 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:35.655917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cpdkp/perf-node-gather-daemonset-hw75x"
Jul 23 18:49:36.747212 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:36.747183    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pc744_cdff1e77-7007-4433-baa9-c4a3276613ad/manager/0.log"
Jul 23 18:49:36.993983 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:36.993286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p5bpm_48f610a2-b455-44a4-8a4e-229a376e4dcc/manager/0.log"
Jul 23 18:49:37.221665 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:37.221614    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86bcf48f9b-ldsm9_91e7e0bb-57d8-4d84-976e-8f92f2f761db/manager/0.log"
Jul 23 18:49:37.242801 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:37.242776    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gngw7_a79a2916-00f0-47b2-b9a2-60bd57c5b596/server/0.log"
Jul 23 18:49:37.344220 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:37.344183    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jg4bl_837026c8-64fa-4afa-b45c-3bbb0d092487/manager/0.log"
Jul 23 18:49:37.599927 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:37.599895    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-wqmd5_f6619904-ed33-46f9-8c3c-95429e190ea9/manager/0.log"
Jul 23 18:49:37.626711 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:37.626674    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657dbb4b4c-wqmd5_f6619904-ed33-46f9-8c3c-95429e190ea9/copy-manifests/0.log"
Jul 23 18:49:38.547871 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:38.547839    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77d9cbf8b4-wmqzj_ffd75b15-abb2-404b-9978-7e1252260b6e/kube-auth-proxy/0.log"
Jul 23 18:49:38.622280 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:38.622242    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rl2d4_f8c4aab0-e19b-4717-98bf-0d0bccfc76e6/payload-pre-processing/0.log"
Jul 23 18:49:38.647576 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:38.647508    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wcvnv_fa981909-b02a-4eb8-aff2-3219f398b491/payload-processing/0.log"
Jul 23 18:49:38.664223 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:38.664194    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7cc9cdfd-b5f4s_aab248f1-1f0e-4550-abfb-ccf8deff11e6/router/0.log"
Jul 23 18:49:40.113966 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:40.113918    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77d9cbf8b4-wmqzj_ffd75b15-abb2-404b-9978-7e1252260b6e/kube-auth-proxy/0.log"
Jul 23 18:49:40.192416 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:40.192337    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-rl2d4_f8c4aab0-e19b-4717-98bf-0d0bccfc76e6/payload-pre-processing/0.log"
Jul 23 18:49:40.218746 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:40.218718    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wcvnv_fa981909-b02a-4eb8-aff2-3219f398b491/payload-processing/0.log"
Jul 23 18:49:40.233288 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:40.233256    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7cc9cdfd-b5f4s_aab248f1-1f0e-4550-abfb-ccf8deff11e6/router/0.log"
Jul 23 18:49:44.164778 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:44.164748    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-nd6rt_a66476a4-ff22-450d-a5d2-d96333402bc6/migrator/0.log"
Jul 23 18:49:44.189493 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:44.189464    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-nd6rt_a66476a4-ff22-450d-a5d2-d96333402bc6/graceful-termination/0.log"
Jul 23 18:49:45.954411 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:45.954379    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lqf6q_067b5e5d-0bd1-41cd-b076-7118f4472717/kube-multus-additional-cni-plugins/0.log"
Jul 23 18:49:45.979063 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:45.979018    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lqf6q_067b5e5d-0bd1-41cd-b076-7118f4472717/egress-router-binary-copy/0.log"
Jul 23 18:49:46.008375 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:46.008348    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lqf6q_067b5e5d-0bd1-41cd-b076-7118f4472717/cni-plugins/0.log"
Jul 23 18:49:46.046048 ip-10-0-130-3 kubenswrapper[2562]: I0723 18:49:46.046019    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lqf6q_067b5e5d-0bd1-41cd-b076-7118f4472717/bond-cni-plugin/0.log"