May 27 23:38:18.452243 ip-10-0-133-162 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 27 23:38:18.452254 ip-10-0-133-162 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 27 23:38:18.452261 ip-10-0-133-162 systemd[1]: kubelet.service: Failed with result 'resources'.
May 27 23:38:18.452474 ip-10-0-133-162 systemd[1]: Failed to start Kubernetes Kubelet.
May 27 23:38:28.557918 ip-10-0-133-162 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 27 23:38:28.557935 ip-10-0-133-162 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3197e6c7049745e4a73a44e256880ac9 --
May 27 23:40:50.479569 ip-10-0-133-162 systemd[1]: Starting Kubernetes Kubelet...
May 27 23:40:50.985580 ip-10-0-133-162 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 23:40:50.985580 ip-10-0-133-162 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 27 23:40:50.985580 ip-10-0-133-162 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 23:40:50.985580 ip-10-0-133-162 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 27 23:40:50.985580 ip-10-0-133-162 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 23:40:50.986443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.986347    2572 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 27 23:40:50.991597 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991576    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:40:50.991597 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991592    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:40:50.991597 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991597    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:40:50.991597 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991601    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:40:50.991597 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991606    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991612    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991616    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991619    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991623    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991627    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991630    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991634    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991637    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991641    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991645    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991650    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991653    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991657    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991661    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991664    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991669    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991672    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991676    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991680    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:40:50.991882 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991684    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991688    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991692    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991696    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991700    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991703    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991708    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991712    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991716    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991720    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991725    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991728    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991733    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991738    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991742    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991748    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991753    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991757    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991761    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991765    2572 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:40:50.992762 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991770    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991774    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991778    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991782    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991787    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991792    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991796    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991801    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991804    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991809    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991813    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991819    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991823    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991828    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991832    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991836    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991840    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991844    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991849    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:40:50.993624 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991854    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991858    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991865    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991872    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991876    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991880    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991884    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991889    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991894    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991899    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991903    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991908    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991913    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991920    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991926    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991930    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991935    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991940    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991945    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:40:50.994363 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991949    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991954    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991959    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.991964    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992601    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992612    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992618    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992623    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992628    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992633    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992638    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992643    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992648    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992652    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992656    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992661    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992665    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992669    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992674    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:40:50.994830 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992678    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992683    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992687    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992692    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992697    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992701    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992705    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992709    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992713    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992717    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992721    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992725    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992732    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992738    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992743    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992748    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992752    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992757    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992761    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:40:50.995277 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992765    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992769    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992773    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992777    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992781    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992785    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992789    2572 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992793    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992797    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992801    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992806    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992810    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992814    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992819    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992823    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992827    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992831    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992837    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992841    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992845    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:40:50.995806 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992849    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992854    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992857    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992862    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992866    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992870    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992874    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992878    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992882    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992887    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992891    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992895    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992900    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992904    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992908    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992912    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992919    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992923    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992927    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992931    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:40:50.996289 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992936    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992940    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992944    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992948    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992954    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992960    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992966    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992970    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992975    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992979    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992984    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.992988    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993096    2572 flags.go:64] FLAG: --address="0.0.0.0"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993107    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993118    2572 flags.go:64] FLAG: --anonymous-auth="true"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993126    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993132    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993137    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993144    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993150    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993156    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 27 23:40:50.996811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993162    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993167    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993173    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993178    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993183    2572 flags.go:64] FLAG: --cgroup-root=""
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993188    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993193    2572 flags.go:64] FLAG: --client-ca-file=""
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993197    2572 flags.go:64] FLAG: --cloud-config=""
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993203    2572 flags.go:64] FLAG: --cloud-provider="external"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993208    2572 flags.go:64] FLAG: --cluster-dns="[]"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993215    2572 flags.go:64] FLAG: --cluster-domain=""
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993219    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993224    2572 flags.go:64] FLAG: --config-dir=""
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993229    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993235    2572 flags.go:64] FLAG: --container-log-max-files="5"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993241    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993246    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993252    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993258    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993262    2572 flags.go:64] FLAG: --contention-profiling="false"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993267    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993272    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993344    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993388    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993397    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 27 23:40:50.997340 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993403    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993408    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993413    2572 flags.go:64] FLAG: --enable-load-reader="false"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993419    2572 flags.go:64] FLAG: --enable-server="true"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993424    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993443    2572 flags.go:64] FLAG: --event-burst="100"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.993452    2572 flags.go:64] FLAG: --event-qps="50"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994568    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994574    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994579    2572 flags.go:64] FLAG: --eviction-hard=""
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994594    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994601    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994607    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994634    2572 flags.go:64] FLAG: --eviction-soft=""
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994741    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994751    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994756    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994761    2572 flags.go:64] FLAG: --experimental-mounter-path=""
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994764    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994768    2572 flags.go:64] FLAG: --fail-swap-on="true"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994771    2572 flags.go:64] FLAG: --feature-gates=""
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994775    2572 flags.go:64] FLAG: --file-check-frequency="20s"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994779    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994782    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994785    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994789    2572 flags.go:64] FLAG: --healthz-port="10248"
May 27 23:40:50.997927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994793    2572 flags.go:64] FLAG: --help="false"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994796    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-133-162.ec2.internal"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994799    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994802    2572 flags.go:64] FLAG: --http-check-frequency="20s"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994805    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994809    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994813    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994816    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994819    2572 flags.go:64] FLAG: --image-service-endpoint=""
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994821    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994824    2572 flags.go:64] FLAG: --kube-api-burst="100"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994829    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994832    2572 flags.go:64] FLAG: --kube-api-qps="50"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994836    2572 flags.go:64] FLAG: --kube-reserved=""
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994839    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994842    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994845    2572 flags.go:64] FLAG: --kubelet-cgroups=""
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994849    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994852    2572 flags.go:64] FLAG: --lock-file=""
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994854    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994858    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994861    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994875    2572 flags.go:64] FLAG: --log-json-split-stream="false"
May 27 23:40:50.998562 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994878    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994881    2572 flags.go:64] FLAG: --log-text-split-stream="false"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994883    2572 flags.go:64] FLAG: --logging-format="text"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994886    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994890    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994893    2572 flags.go:64] FLAG: --manifest-url=""
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994895    2572 flags.go:64] FLAG: --manifest-url-header=""
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994900    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994904    2572 flags.go:64] FLAG: --max-open-files="1000000"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994908    2572 flags.go:64] FLAG: --max-pods="110"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994912    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994915    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994918    2572 flags.go:64] FLAG: --memory-manager-policy="None"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994921    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994924    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994927    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994930    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994939    2572 flags.go:64] FLAG: --node-status-max-images="50"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994943    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994946    2572 flags.go:64] FLAG: --oom-score-adj="-999"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994949    2572 flags.go:64] FLAG: --pod-cidr=""
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994952    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994958    2572 flags.go:64] FLAG: --pod-manifest-path=""
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994961    2572 flags.go:64] FLAG: --pod-max-pids="-1"
May 27 23:40:50.999104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994965    2572 flags.go:64] FLAG: --pods-per-core="0"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994968    2572 flags.go:64] FLAG: --port="10250"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994971    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994974    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05906ee6a906f0b90"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994977    2572 flags.go:64] FLAG: --qos-reserved=""
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994980    2572 flags.go:64] FLAG: --read-only-port="10255"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994983    2572 flags.go:64] FLAG: --register-node="true"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994985    2572 flags.go:64] FLAG: --register-schedulable="true"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994988    2572 flags.go:64] FLAG: --register-with-taints=""
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994992    2572 flags.go:64] FLAG: --registry-burst="10"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994995    2572 flags.go:64] FLAG: --registry-qps="5"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.994997    2572 flags.go:64] FLAG: --reserved-cpus=""
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995000    2572 flags.go:64] FLAG: --reserved-memory=""
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995004    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995007    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995010    2572 flags.go:64] FLAG: --rotate-certificates="false"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995013    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995016    2572 flags.go:64] FLAG: --runonce="false"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995019    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995022    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995025    2572 flags.go:64] FLAG: --seccomp-default="false"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995028    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995031    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995034    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995037    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995040    2572 flags.go:64] FLAG: --storage-driver-password="root"
May 27 23:40:50.999713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995042    2572 flags.go:64] FLAG: --storage-driver-secure="false"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995045    2572 flags.go:64] FLAG: --storage-driver-table="stats"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995048    2572 flags.go:64] FLAG: --storage-driver-user="root"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995051    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995055    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995058    2572 flags.go:64] FLAG: --system-cgroups=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995061    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995067    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995070    2572 flags.go:64] FLAG: --tls-cert-file=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995073    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995078    2572 flags.go:64] FLAG: --tls-min-version=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995081    2572 flags.go:64] FLAG: --tls-private-key-file=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995084    2572 flags.go:64] FLAG: --topology-manager-policy="none"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995087    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995090    2572 flags.go:64] FLAG: --topology-manager-scope="container"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995093    2572 flags.go:64] FLAG: --v="2"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995098    2572 flags.go:64] FLAG: --version="false"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995103    2572 flags.go:64] FLAG: --vmodule=""
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995108    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.995111    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995204    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995208    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995211    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995214    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:40:51.000326 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995217    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995220    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995223    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995226    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995228    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995231    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995233    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995236    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995239    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995241    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995244    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995246    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995253    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995256    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995259    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995262    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995265    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995268    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995271    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995274    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:40:51.000956 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995277    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995279    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995282    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995285    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995287    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995290    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995293    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995296    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995300    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995320    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995324    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995327    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995330    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995332    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995335    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995338    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995341    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995344    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995346    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:40:51.001473 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995349    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995352    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995354    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995357    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995359    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995364    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995367    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995370    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995372    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995375    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995378    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995380    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995384    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995386    2572 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995389    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995391    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995394    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995397    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995399    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:40:51.002050 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995402    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995404    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995407    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995409    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995413    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995417    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995419    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995422    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995424    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995427    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995430    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995432    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995435    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995438    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995440    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995443    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995445    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995448    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995452    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995455    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:40:51.002539 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995458    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:40:51.003014 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995460    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:40:51.003014 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995463    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:40:51.003014 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:50.995465    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:40:51.003014 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:50.996292    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.003024    2572 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.003041    2572 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003087    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003092    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003096    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003099    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003102    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003105    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003108    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003110    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003113    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003116    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003119    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003121    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:40:51.003118 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003125    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003127    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003130    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003133    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003135    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003138    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003140    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003143    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003146    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003149    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003151    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003154    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003156    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003159    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003161    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003164    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003166    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003169    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003171    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003175    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:40:51.003524 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003178    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003180    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003183    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003185    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003188    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003190    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003195    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003200    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003203    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003206    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003208    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003211    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003214    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003217    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003220    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003223    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003227    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003229    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003232    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:40:51.004004 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003235    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003238    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003240    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003243    2572 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003245    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003248    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003250    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003253    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003255    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003259    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003263    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003266    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003268    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003271    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003274    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003276    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003279    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003281    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003284    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003287    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:40:51.004506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003289    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003292    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003294    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003297    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003300    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003319    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003324    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003327    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003329    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003332    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003334    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003337    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003340    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003343    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003346    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:40:51.004992 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.003351    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003453    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003458    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003462    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003465    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003468    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003471    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003474    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003477    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003479    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003482    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003486    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003488    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003491    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003494    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003496    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003499    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003501    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003504    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 23:40:51.005382 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003507    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003510    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003515    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003518    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003521    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003524    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003527    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003531    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003535    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003538    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003541    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003544    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003547    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003549    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003552    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003554    2572 feature_gate.go:328] unrecognized feature gate: Example
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003557    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003559    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003562    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 23:40:51.005835 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003564    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003567    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003569    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003572    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003574    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003577    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003580    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003583    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003585    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003588    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003590    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003593    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003595    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003597    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003600    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003602    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003605    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003607    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003610    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003613    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 23:40:51.006282 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003615    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003618    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003620    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003623    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003625    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003628    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003630    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003633    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003635    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003637    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003640    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003642    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003645    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003647    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003650    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003652    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003654    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003657    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003659    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003663    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 23:40:51.006789 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003665    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003667    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003670    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003672    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003675    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003677    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003680    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003683    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:51.003685    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.003690    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 23:40:51.007268 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.004430    2572 server.go:962] "Client rotation is on, will bootstrap in background"
May 27 23:40:51.008096 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.008081    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 27 23:40:51.009169 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.009157    2572 server.go:1019] "Starting client certificate rotation"
May 27 23:40:51.009282 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.009265    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 27 23:40:51.009359 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.009331    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 27 23:40:51.037478 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.037457    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 27 23:40:51.039382 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.039361    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 27 23:40:51.057050 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.057026    2572 log.go:25] "Validated CRI v1 runtime API"
May 27 23:40:51.062799 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.062780    2572 log.go:25] "Validated CRI v1 image API"
May 27 23:40:51.064202 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.064186    2572 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 27 23:40:51.071248 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.071230    2572 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 27 23:40:51.071576 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.071558    2572 fs.go:135] Filesystem UUIDs: map[1043184e-f394-4f45-a341-d68455dd14d8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 91a0d7fc-2eb8-451c-b2a0-a488cb61e6e8:/dev/nvme0n1p4]
May 27 23:40:51.071613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.071578    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 27 23:40:51.075935 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.075828    2572 manager.go:217] Machine: {Timestamp:2026-05-27 23:40:51.075331313 +0000 UTC m=+0.464835117 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098242 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b6f5137022eb68e5822122804bf45 SystemUUID:ec2b6f51-3702-2eb6-8e58-22122804bf45 BootID:3197e6c7-0497-45e4-a73a-44e256880ac9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:3f:56:66:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:3f:56:66:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:4b:f2:51:c3:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 27 23:40:51.076645 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.076634    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 27 23:40:51.076728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.076716    2572 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 27 23:40:51.077870 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.077850    2572 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 27 23:40:51.077999 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.077872    2572 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 27 23:40:51.078048 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.078008    2572 topology_manager.go:138] "Creating topology manager with none policy"
May 27 23:40:51.078048 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.078017    2572 container_manager_linux.go:306] "Creating device plugin manager"
May 27 23:40:51.078048 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.078024    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 27 23:40:51.078864 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.078853    2572 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 27 23:40:51.081137 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.081126    2572 state_mem.go:36] "Initialized new in-memory state store"
May 27 23:40:51.081246 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.081237    2572 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 27 23:40:51.085118 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.085106    2572 kubelet.go:491] "Attempting to sync node with API server"
May 27 23:40:51.085163 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.085124    2572 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 27 23:40:51.085163 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.085135    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 27 23:40:51.085163 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.085143    2572 kubelet.go:397] "Adding apiserver pod source"
May 27 23:40:51.085163 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.085152    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 27 23:40:51.086386 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.086373    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 27 23:40:51.086424 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.086400    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 27 23:40:51.090414 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.090399    2572 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 27 23:40:51.092441 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.092420    2572 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 27 23:40:51.093484 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093468    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093488    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093499    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093507    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093520    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093529    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093537    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093545    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093555    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 27 23:40:51.093572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093564    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 27 23:40:51.093814 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093578    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 27 23:40:51.093994 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.093983    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 27 23:40:51.094943 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.094932    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 27 23:40:51.094995 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.094949    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 27 23:40:51.098215 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098194    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbvz6"
May 27 23:40:51.098366 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098353    2572 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 27 23:40:51.098437 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.098371    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 27 23:40:51.098437 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098378    2572 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 27 23:40:51.098437 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098388    2572 server.go:1295] "Started kubelet"
May 27 23:40:51.098437 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.098391    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 27 23:40:51.098437 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098421    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 27 23:40:51.098627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098501    2572 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 27 23:40:51.098627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.098558    2572 server_v1.go:47] "podresources" method="list" useActivePods=true
May 27 23:40:51.100828 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.100801    2572 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 27 23:40:51.101048 ip-10-0-133-162 systemd[1]: Started Kubernetes Kubelet.
May 27 23:40:51.102830 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.102295    2572 server.go:317] "Adding debug handlers to kubelet server"
May 27 23:40:51.106415 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.106397    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbvz6"
May 27 23:40:51.109485 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.108363    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-162.ec2.internal.18b39095de774faf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-162.ec2.internal,UID:ip-10-0-133-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-162.ec2.internal,},FirstTimestamp:2026-05-27 23:40:51.098365871 +0000 UTC m=+0.487869680,LastTimestamp:2026-05-27 23:40:51.098365871 +0000 UTC m=+0.487869680,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-162.ec2.internal,}"
May 27 23:40:51.111046 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.111028    2572 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 27 23:40:51.111593 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111572    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 27 23:40:51.111593 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111585    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 27 23:40:51.111736 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111626    2572 volume_manager.go:295] "The desired_state_of_world populator starts"
May 27 23:40:51.111736 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111635    2572 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 27 23:40:51.111834 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111756    2572 reconstruct.go:97] "Volume reconstruction finished"
May 27 23:40:51.111834 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111766    2572 reconciler.go:26] "Reconciler: start to sync state"
May 27 23:40:51.111969 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.111952    2572 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 27 23:40:51.112461 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112445    2572 factory.go:55] Registering systemd factory
May 27 23:40:51.112560 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112470    2572 factory.go:223] Registration of the systemd container factory successfully
May 27 23:40:51.112664 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.112646    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.112717 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112689    2572 factory.go:153] Registering CRI-O factory
May 27 23:40:51.112717 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112700    2572 factory.go:223] Registration of the crio container factory successfully
May 27 23:40:51.112799 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112740    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 27 23:40:51.112799 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112753    2572 factory.go:103] Registering Raw factory
May 27 23:40:51.112799 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.112763    2572 manager.go:1196] Started watching for new ooms in manager
May 27 23:40:51.113141 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.113026    2572 manager.go:319] Starting recovery of all containers
May 27 23:40:51.123175 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.123149    2572 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:51.125748 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.125726    2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-162.ec2.internal\" not found" node="ip-10-0-133-162.ec2.internal"
May 27 23:40:51.125984 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.125971    2572 manager.go:324] Recovery completed
May 27 23:40:51.129803 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.129790    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:40:51.131672 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.131647    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:40:51.131746 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.131677    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:40:51.131746 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.131688    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID"
May 27 23:40:51.132122 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.132106    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 27 23:40:51.132194 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.132128    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 27 23:40:51.132194 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.132149    2572 state_mem.go:36] "Initialized new in-memory state store"
May 27 23:40:51.136938 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.136923    2572 policy_none.go:49] "None policy: Start"
May 27 23:40:51.137009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.136944    2572 memory_manager.go:186] "Starting memorymanager" policy="None"
May 27 23:40:51.137009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.136957    2572 state_mem.go:35] "Initializing new in-memory state store"
May 27 23:40:51.181879 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.181855    2572 manager.go:341] "Starting Device Plugin manager"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.181896    2572 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.181910    2572 server.go:85] "Starting device plugin registration server"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.182124    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.182139    2572 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.182216    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.182329    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.182340    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.182824    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 27 23:40:51.195120 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.182859    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.213574 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.213542    2572 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 27 23:40:51.214879 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.214859    2572 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 27 23:40:51.214879 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.214882    2572 status_manager.go:230] "Starting to sync pod status with apiserver"
May 27 23:40:51.215008 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.214897    2572 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 27 23:40:51.215008 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.214904    2572 kubelet.go:2451] "Starting kubelet main sync loop"
May 27 23:40:51.215008 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.214935    2572 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 27 23:40:51.221749 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.221729    2572 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:51.282832 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.282788    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:40:51.283760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.283736    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:40:51.283822 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.283771    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:40:51.283822 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.283784    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID"
May 27 23:40:51.283822 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.283805    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-162.ec2.internal"
May 27 23:40:51.290042 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.290027    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-162.ec2.internal"
May 27 23:40:51.290085 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.290049    2572 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-162.ec2.internal\": node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.307893 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.307871    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.315182 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.315167    2572 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"]
May 27 23:40:51.315244 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.315217    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:40:51.315942 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.315927    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:40:51.316019 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.315958    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:40:51.316019 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.315971    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID"
May 27 23:40:51.317258 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317234    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:40:51.317412 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317398    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.317450 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317428    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:40:51.317930 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317912    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:40:51.317930 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317921    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:40:51.318004 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317936    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:40:51.318004 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317943    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:40:51.318004 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317945    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID"
May 27 23:40:51.318004 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.317954    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID"
May 27 23:40:51.319124 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.319101    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.319124 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.319125    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 23:40:51.319738 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.319723    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory"
May 27 23:40:51.319817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.319746    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure"
May 27 23:40:51.319817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.319762    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID"
May 27 23:40:51.345837 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.345820    2572 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-162.ec2.internal\" not found" node="ip-10-0-133-162.ec2.internal"
May 27 23:40:51.350210 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.350197    2572 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-162.ec2.internal\" not found" node="ip-10-0-133-162.ec2.internal"
May 27 23:40:51.407955 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.407928    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.508184 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.508151    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.512408 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.512387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a329a27d84f7b98ef3b36fc9a735cfdf-config\") pod \"kube-apiserver-proxy-ip-10-0-133-162.ec2.internal\" (UID: \"a329a27d84f7b98ef3b36fc9a735cfdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.512462 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.512419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae9a7b48520c58f68b30a66c3e150b8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"ae9a7b48520c58f68b30a66c3e150b8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.512462 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.512439    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9a7b48520c58f68b30a66c3e150b8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"ae9a7b48520c58f68b30a66c3e150b8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.608801 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.608739    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.613020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.613005    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae9a7b48520c58f68b30a66c3e150b8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"ae9a7b48520c58f68b30a66c3e150b8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.613097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.613037    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9a7b48520c58f68b30a66c3e150b8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"ae9a7b48520c58f68b30a66c3e150b8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.613097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.613062    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a329a27d84f7b98ef3b36fc9a735cfdf-config\") pod \"kube-apiserver-proxy-ip-10-0-133-162.ec2.internal\" (UID: \"a329a27d84f7b98ef3b36fc9a735cfdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.613097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.613063    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae9a7b48520c58f68b30a66c3e150b8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"ae9a7b48520c58f68b30a66c3e150b8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.613097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.613086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a329a27d84f7b98ef3b36fc9a735cfdf-config\") pod \"kube-apiserver-proxy-ip-10-0-133-162.ec2.internal\" (UID: \"a329a27d84f7b98ef3b36fc9a735cfdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.613230 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.613092    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9a7b48520c58f68b30a66c3e150b8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"ae9a7b48520c58f68b30a66c3e150b8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.649114 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.649098    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.652789 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.652771    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"
May 27 23:40:51.709312 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.709289    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.809705 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.809676    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.909937 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:51.909906    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:51.910087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:51.909953    2572 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:52.008740 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.008707    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 27 23:40:52.009424 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.008864    2572 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 27 23:40:52.009424 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.008902    2572 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 27 23:40:52.009424 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.008902    2572 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 27 23:40:52.010863 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:52.010845    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:52.110393 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.110360    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-26 23:35:51 +0000 UTC" deadline="2028-03-09 17:45:53.567644596 +0000 UTC"
May 27 23:40:52.110393 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.110389    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15642h5m1.457258676s"
May 27 23:40:52.111456 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:52.111432    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found"
May 27 23:40:52.112530 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.112518    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 27 23:40:52.128523 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.128497    2572 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 27 23:40:52.164885 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.164829    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gfq8"
May 27 23:40:52.174824 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.174806    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gfq8"
May 27 23:40:52.183362 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.183342    2572 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:52.205174 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:52.205148    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9a7b48520c58f68b30a66c3e150b8c.slice/crio-139edb292af61a502f6671436cb54117ef098dbf637f5d107d4e5020e01eb2cf WatchSource:0}: Error finding container 139edb292af61a502f6671436cb54117ef098dbf637f5d107d4e5020e01eb2cf: Status 404 returned error can't find the container with id 139edb292af61a502f6671436cb54117ef098dbf637f5d107d4e5020e01eb2cf
May 27 23:40:52.205376 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:52.205362    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda329a27d84f7b98ef3b36fc9a735cfdf.slice/crio-4207f9d4a133085c569d2bce4cc09a7daace5b7ac00fd499eb284a188666e916 WatchSource:0}: Error finding container 4207f9d4a133085c569d2bce4cc09a7daace5b7ac00fd499eb284a188666e916: Status 404 returned error can't find the container with id 4207f9d4a133085c569d2bce4cc09a7daace5b7ac00fd499eb284a188666e916
May 27 23:40:52.209052 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.209038    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:40:52.211874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.211853    2572 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal"
May 27 23:40:52.217461 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.217423    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" event={"ID":"a329a27d84f7b98ef3b36fc9a735cfdf","Type":"ContainerStarted","Data":"4207f9d4a133085c569d2bce4cc09a7daace5b7ac00fd499eb284a188666e916"}
May 27 23:40:52.218187 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.218166    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" event={"ID":"ae9a7b48520c58f68b30a66c3e150b8c","Type":"ContainerStarted","Data":"139edb292af61a502f6671436cb54117ef098dbf637f5d107d4e5020e01eb2cf"}
May 27 23:40:52.230050 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.230028    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 27 23:40:52.233935 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.233922    2572 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"
May 27 23:40:52.243554 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.243541    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 27 23:40:52.921517 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:52.921367    2572 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:53.085537 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.085509    2572 apiserver.go:52] "Watching apiserver"
May 27 23:40:53.093252 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.093219    2572 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 27 23:40:53.093638 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.093613    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-hzvkg","openshift-dns/node-resolver-flbbx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal","openshift-multus/multus-additional-cni-plugins-rzrfc","openshift-network-diagnostics/network-check-target-ttrwc","openshift-network-operator/iptables-alerter-n58tk","kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7","openshift-cluster-node-tuning-operator/tuned-4wvc2","openshift-image-registry/node-ca-lw89q","openshift-multus/multus-pjwfd","openshift-multus/network-metrics-daemon-bp6lz","openshift-ovn-kubernetes/ovnkube-node-cpqgv"]
May 27 23:40:53.096137 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.096113    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.098191 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.098168    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.098677 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.098656    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 27 23:40:53.098765 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.098720    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 27 23:40:53.098842 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.098822    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wh5j\""
May 27 23:40:53.099033 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.099017    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 27 23:40:53.100453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.100433    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m8ppk\""
May 27 23:40:53.100604 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.100589    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 27 23:40:53.101188 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.101169    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 27 23:40:53.102383 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.102367    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.104892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.104471    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 27 23:40:53.104892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.104556    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 27 23:40:53.104892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.104686    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 27 23:40:53.104892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.104708    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m9bc8\""
May 27 23:40:53.104892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.104720    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 27 23:40:53.104892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.104726    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 27 23:40:53.105292 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.105276    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:53.105402 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.105360    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:40:53.107966 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.107947    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.110139 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.110121    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.110912 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.110895    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 27 23:40:53.110912 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.110908    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 27 23:40:53.111051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.110917    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 27 23:40:53.111051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.110989    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-njjfs\""
May 27 23:40:53.112443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.112200    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 27 23:40:53.112443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.112200    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 27 23:40:53.112443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.112272    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 27 23:40:53.112443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.112428    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.113532 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.113442    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4nqp6\""
May 27 23:40:53.114287 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.114270    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 27 23:40:53.114287 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.114280    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62cjm\""
May 27 23:40:53.114923 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.114665    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.114923 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.114780    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.115043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.114932    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 27 23:40:53.116708 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.116689    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vr4bc\""
May 27 23:40:53.116794 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.116689    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w8dls\""
May 27 23:40:53.116794 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.116762    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 27 23:40:53.116900 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.116689    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 27 23:40:53.116991 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.116974    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 27 23:40:53.117123 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.117107    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.117187 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.117171    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:40:53.119251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bba202-befe-4b9a-9ca9-88e58a64a498-tmp\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.119381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119273    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-hosts-file\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.119381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-system-cni-dir\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.119381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cnibin\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.119536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119388    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.119536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-sys-fs\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.119536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119454    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.119536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119488    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bscx\" (UniqueName: \"kubernetes.io/projected/0cdb4626-9749-4c04-9159-2c63485fe159-kube-api-access-6bscx\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.119536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0616e250-c245-4aa5-aaaa-ae15d9e3234e-serviceca\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-device-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-tuned\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzl4h\" (UniqueName: \"kubernetes.io/projected/b2bba202-befe-4b9a-9ca9-88e58a64a498-kube-api-access-hzl4h\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.119767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysconfig\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119776    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-systemd\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-lib-modules\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0cdb4626-9749-4c04-9159-2c63485fe159-host-slash\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0616e250-c245-4aa5-aaaa-ae15d9e3234e-host\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9n9\" (UniqueName: \"kubernetes.io/projected/0616e250-c245-4aa5-aaaa-ae15d9e3234e-kube-api-access-jr9n9\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-kubernetes\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d28d858-235d-4e96-9f9a-57634afaa5c9-agent-certs\") pod \"konnectivity-agent-hzvkg\" (UID: \"3d28d858-235d-4e96-9f9a-57634afaa5c9\") " pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.119996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-tmp-dir\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120060    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0cdb4626-9749-4c04-9159-2c63485fe159-iptables-alerter-script\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.120087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-socket-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.120590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-modprobe-d\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-run\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-var-lib-kubelet\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d28d858-235d-4e96-9f9a-57634afaa5c9-konnectivity-ca\") pod \"konnectivity-agent-hzvkg\" (UID: \"3d28d858-235d-4e96-9f9a-57634afaa5c9\") " pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.120590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gf4\" (UniqueName: \"kubernetes.io/projected/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-kube-api-access-c8gf4\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.120590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120559    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-os-release\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.120766 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.120766 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120616    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t49w\" (UniqueName: \"kubernetes.io/projected/a6462972-dbcb-47e2-85b2-a43fac1480f8-kube-api-access-9t49w\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.120766 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-sys\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120766 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-host\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.120766 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.120680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-registration-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.121227 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.121158    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysctl-conf\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.121227 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.121218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrl6\" (UniqueName: \"kubernetes.io/projected/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-kube-api-access-jfrl6\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.121400 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.121275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysctl-d\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.122140 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.122121    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.124587 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.124443    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 27 23:40:53.124587 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.124470    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 27 23:40:53.124587 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.124481    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 27 23:40:53.124587 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.124495    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 27 23:40:53.124587 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.124474    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 27 23:40:53.124587 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.124501    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bbs7k\""
May 27 23:40:53.125464 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.125447    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 27 23:40:53.176125 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.176054    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-26 23:35:52 +0000 UTC" deadline="2028-03-09 21:47:30.436672448 +0000 UTC"
May 27 23:40:53.176125 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.176079    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15646h6m37.260595825s"
May 27 23:40:53.213606 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.213580    2572 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 27 23:40:53.221572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-etc-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.221706 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221590    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrl6\" (UniqueName: \"kubernetes.io/projected/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-kube-api-access-jfrl6\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.221706 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221653    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-cni-netd\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.221706 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221693    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsk4\" (UniqueName: \"kubernetes.io/projected/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-kube-api-access-xfsk4\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221730    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cnibin\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221752    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221777    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-cnibin\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-netns\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221792    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cnibin\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221814    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-systemd-units\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221852    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-ovn\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.221891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221874    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221920    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bscx\" (UniqueName: \"kubernetes.io/projected/0cdb4626-9749-4c04-9159-2c63485fe159-kube-api-access-6bscx\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221943    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0616e250-c245-4aa5-aaaa-ae15d9e3234e-serviceca\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.221984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovn-node-metrics-cert\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222019    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222041    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222066    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222089    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-systemd\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222111    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-lib-modules\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-cni-multus\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-slash\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222184    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0cdb4626-9749-4c04-9159-2c63485fe159-host-slash\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222206    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0616e250-c245-4aa5-aaaa-ae15d9e3234e-host\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222205    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.222275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0616e250-c245-4aa5-aaaa-ae15d9e3234e-host\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222568    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9n9\" (UniqueName: \"kubernetes.io/projected/0616e250-c245-4aa5-aaaa-ae15d9e3234e-kube-api-access-jr9n9\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222615    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-kubernetes\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-node-log\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222706    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222712    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-tmp-dir\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222752    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-lib-modules\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-cni-binary-copy\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222832    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222839    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0616e250-c245-4aa5-aaaa-ae15d9e3234e-serviceca\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.222874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222850    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d28d858-235d-4e96-9f9a-57634afaa5c9-konnectivity-ca\") pod \"konnectivity-agent-hzvkg\" (UID: \"3d28d858-235d-4e96-9f9a-57634afaa5c9\") " pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222882    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-os-release\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222915    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0cdb4626-9749-4c04-9159-2c63485fe159-host-slash\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222919    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222969    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9t49w\" (UniqueName: \"kubernetes.io/projected/a6462972-dbcb-47e2-85b2-a43fac1480f8-kube-api-access-9t49w\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.222989    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-kubelet\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223020    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-registration-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-os-release\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovnkube-config\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223077    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-env-overrides\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223129    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysctl-d\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223174    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.223207 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223186    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-etc-kubernetes\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223223    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223261    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-hosts-file\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223296    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-system-cni-dir\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223322    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysctl-d\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223363    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-sys-fs\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223384    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-systemd\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223402    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-system-cni-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.223627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.223881 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223677    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-registration-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.223881 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223786    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-kubernetes\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.223881 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223806    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-device-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.223881 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223843    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-tuned\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.223881 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223871    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-system-cni-dir\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.224043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223888    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-sys-fs\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.224043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223895    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzl4h\" (UniqueName: \"kubernetes.io/projected/b2bba202-befe-4b9a-9ca9-88e58a64a498-kube-api-access-hzl4h\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.224043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223928    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.224043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-run-netns\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.224043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223966    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-log-socket\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.224043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.223993    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-cni-bin\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.224238 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224172    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-tmp-dir\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.224278 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224247    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysconfig\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.224339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224287    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-socket-dir-parent\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.224339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224324    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 27 23:40:53.224426 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224363    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-daemon-config\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.224426 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224409    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-hostroot\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.224489 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-var-lib-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.224699 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.224675    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-hosts-file\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.225093 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovnkube-script-lib\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.225210 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkvt\" (UniqueName: \"kubernetes.io/projected/6fe3b412-20be-435b-9aaa-ef0602a81cd5-kube-api-access-lvkvt\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.225210 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225150    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d28d858-235d-4e96-9f9a-57634afaa5c9-agent-certs\") pod \"konnectivity-agent-hzvkg\" (UID: \"3d28d858-235d-4e96-9f9a-57634afaa5c9\") " pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.225210 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225177    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0cdb4626-9749-4c04-9159-2c63485fe159-iptables-alerter-script\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.225377 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225210    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-socket-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.225377 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225241    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-modprobe-d\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225377 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225274    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-run\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225377 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225319    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-var-lib-kubelet\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225377 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225348    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bba202-befe-4b9a-9ca9-88e58a64a498-tmp\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225610 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-cni-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.225610 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225408    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gf4\" (UniqueName: \"kubernetes.io/projected/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-kube-api-access-c8gf4\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.225610 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225580    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-sys\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225610 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225606    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-host\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225758 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225638    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-kubelet\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.225758 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225669    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysctl-conf\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.225758 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225701    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-k8s-cni-cncf-io\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.225758 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-cni-bin\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.225758 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-conf-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.225895 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225784    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-multus-certs\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.225895 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67kp\" (UniqueName: \"kubernetes.io/projected/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-kube-api-access-t67kp\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.225895 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.225841    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-systemd\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.226162 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226143    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-run\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.226235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226198    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-socket-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.226235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226203    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-var-lib-kubelet\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.226788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226680    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d28d858-235d-4e96-9f9a-57634afaa5c9-konnectivity-ca\") pod \"konnectivity-agent-hzvkg\" (UID: \"3d28d858-235d-4e96-9f9a-57634afaa5c9\") " pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.226788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226700    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6462972-dbcb-47e2-85b2-a43fac1480f8-device-dir\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.226788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-os-release\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.226788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226737    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-host\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.226788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226808    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysconfig\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.226788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226843    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-sys\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.227179 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226869    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-sysctl-conf\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.227179 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.226912    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-modprobe-d\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.227491 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.227459    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0cdb4626-9749-4c04-9159-2c63485fe159-iptables-alerter-script\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.229071 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.229027    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bba202-befe-4b9a-9ca9-88e58a64a498-etc-tuned\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.229256 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.229235    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bba202-befe-4b9a-9ca9-88e58a64a498-tmp\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.230943 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.230921    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d28d858-235d-4e96-9f9a-57634afaa5c9-agent-certs\") pod \"konnectivity-agent-hzvkg\" (UID: \"3d28d858-235d-4e96-9f9a-57634afaa5c9\") " pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.239116 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.239093    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:40:53.239231 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.239118    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:40:53.239231 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.239134    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:53.239231 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.239212    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:40:53.739180786 +0000 UTC m=+3.128684604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:53.241408 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.241383    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t49w\" (UniqueName: \"kubernetes.io/projected/a6462972-dbcb-47e2-85b2-a43fac1480f8-kube-api-access-9t49w\") pod \"aws-ebs-csi-driver-node-jjwj7\" (UID: \"a6462972-dbcb-47e2-85b2-a43fac1480f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.241889 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.241864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzl4h\" (UniqueName: \"kubernetes.io/projected/b2bba202-befe-4b9a-9ca9-88e58a64a498-kube-api-access-hzl4h\") pod \"tuned-4wvc2\" (UID: \"b2bba202-befe-4b9a-9ca9-88e58a64a498\") " pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.242059 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.242043    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9n9\" (UniqueName: \"kubernetes.io/projected/0616e250-c245-4aa5-aaaa-ae15d9e3234e-kube-api-access-jr9n9\") pod \"node-ca-lw89q\" (UID: \"0616e250-c245-4aa5-aaaa-ae15d9e3234e\") " pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.242725 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.242706    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gf4\" (UniqueName: \"kubernetes.io/projected/8b3785e5-1e3a-4306-b7cb-64acde7e65ed-kube-api-access-c8gf4\") pod \"node-resolver-flbbx\" (UID: \"8b3785e5-1e3a-4306-b7cb-64acde7e65ed\") " pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.242785 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.242752    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bscx\" (UniqueName: \"kubernetes.io/projected/0cdb4626-9749-4c04-9159-2c63485fe159-kube-api-access-6bscx\") pod \"iptables-alerter-n58tk\" (UID: \"0cdb4626-9749-4c04-9159-2c63485fe159\") " pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.245253 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.245229    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrl6\" (UniqueName: \"kubernetes.io/projected/8ff4dcda-d905-444a-88cc-49ab1ce8b7bf-kube-api-access-jfrl6\") pod \"multus-additional-cni-plugins-rzrfc\" (UID: \"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf\") " pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.265422 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.265401    2572 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:53.326523 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326486    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-run-netns\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326531    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-log-socket\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326561    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-cni-bin\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326591    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-socket-dir-parent\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326608    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-run-netns\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326611    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-log-socket\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326617    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-daemon-config\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.326668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326654    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-cni-bin\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326692    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-hostroot\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326714    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-var-lib-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-socket-dir-parent\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326736    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovnkube-script-lib\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326759    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkvt\" (UniqueName: \"kubernetes.io/projected/6fe3b412-20be-435b-9aaa-ef0602a81cd5-kube-api-access-lvkvt\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326769    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-hostroot\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-var-lib-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326789    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-cni-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326811    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-kubelet\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326831    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-k8s-cni-cncf-io\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326847    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-cni-bin\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326863    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-conf-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326877    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-cni-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326881    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-multus-certs\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326914    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-multus-certs\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326923    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t67kp\" (UniqueName: \"kubernetes.io/projected/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-kube-api-access-t67kp\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326942    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-kubelet\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327009 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326948    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-systemd\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326972    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-etc-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326995    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-k8s-cni-cncf-io\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.326998    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-cni-netd\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327023    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-cni-bin\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327022    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsk4\" (UniqueName: \"kubernetes.io/projected/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-kube-api-access-xfsk4\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327067    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-conf-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-systemd\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-cni-netd\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327135    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-cnibin\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327159    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-netns\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327184    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-systemd-units\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327189    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-multus-daemon-config\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327073    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-etc-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327210    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-ovn\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327239    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-ovn\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327246    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327249    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-run-netns\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.327840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327220    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-cnibin\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327265    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovn-node-metrics-cert\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327268    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-systemd-units\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327287    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327287    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327334    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-cni-multus\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327355    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-slash\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327373    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-node-log\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.327381    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327383    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovnkube-script-lib\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327397    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-cni-binary-copy\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327428    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-slash\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327441    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-kubelet\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327473    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-os-release\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.327496    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:40:53.82747626 +0000 UTC m=+3.216980064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327525    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovnkube-config\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327528    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-node-log\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.328463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327536    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-os-release\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327474    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-cni-multus\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327553    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-env-overrides\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327579    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-etc-kubernetes\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327580    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-host-var-lib-kubelet\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327616    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327646    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-system-cni-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327671    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327731    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327735    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6fe3b412-20be-435b-9aaa-ef0602a81cd5-run-openvswitch\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327780    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-system-cni-dir\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327788    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-cni-binary-copy\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327790    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-etc-kubernetes\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.327975    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-env-overrides\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.328000    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovnkube-config\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.329899 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.329879    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fe3b412-20be-435b-9aaa-ef0602a81cd5-ovn-node-metrics-cert\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.336567 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.336544    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkvt\" (UniqueName: \"kubernetes.io/projected/6fe3b412-20be-435b-9aaa-ef0602a81cd5-kube-api-access-lvkvt\") pod \"ovnkube-node-cpqgv\" (UID: \"6fe3b412-20be-435b-9aaa-ef0602a81cd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.337817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.337791    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsk4\" (UniqueName: \"kubernetes.io/projected/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-kube-api-access-xfsk4\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.338508 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.338483    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67kp\" (UniqueName: \"kubernetes.io/projected/69ebaa5a-f129-4be0-b4a7-3dc11eafd58c-kube-api-access-t67kp\") pod \"multus-pjwfd\" (UID: \"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c\") " pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.406414 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.406379    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lw89q"
May 27 23:40:53.415194 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.415171    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flbbx"
May 27 23:40:53.424791 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.424767    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzrfc"
May 27 23:40:53.430560 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.430516    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n58tk"
May 27 23:40:53.436511 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.436492    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7"
May 27 23:40:53.443058 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.443042    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wvc2"
May 27 23:40:53.449520 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.449499    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:40:53.458016 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.457994    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pjwfd"
May 27 23:40:53.464596 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.464573    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:40:53.830788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.830712    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:53.830788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.830755    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:53.830951 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.830866    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:40:53.830951 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.830867    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:53.830951 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.830879    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:40:53.830951 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.830893    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:53.830951 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.830932    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:40:54.830919446 +0000 UTC m=+4.220423252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:53.830951 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:53.830945    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:40:54.83093849 +0000 UTC m=+4.220442282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:53.858517 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.858488    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ebaa5a_f129_4be0_b4a7_3dc11eafd58c.slice/crio-441215dd6e12fb5eb70d75ca04df9cf50e3f7677177b6e61ac7ebd51819dddb7 WatchSource:0}: Error finding container 441215dd6e12fb5eb70d75ca04df9cf50e3f7677177b6e61ac7ebd51819dddb7: Status 404 returned error can't find the container with id 441215dd6e12fb5eb70d75ca04df9cf50e3f7677177b6e61ac7ebd51819dddb7
May 27 23:40:53.859801 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:53.859721    2572 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 23:40:53.860554 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.860535    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6462972_dbcb_47e2_85b2_a43fac1480f8.slice/crio-f134e1895d59993f3f19d3d60e2a7147764a22576d6444ef2ed9315527bf7eb2 WatchSource:0}: Error finding container f134e1895d59993f3f19d3d60e2a7147764a22576d6444ef2ed9315527bf7eb2: Status 404 returned error can't find the container with id f134e1895d59993f3f19d3d60e2a7147764a22576d6444ef2ed9315527bf7eb2
May 27 23:40:53.862253 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.862224    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3785e5_1e3a_4306_b7cb_64acde7e65ed.slice/crio-1d70d1c51b4b77bbd06f19f3c52015cbb2eada4e2c3d33f84821787cf0def66c WatchSource:0}: Error finding container 1d70d1c51b4b77bbd06f19f3c52015cbb2eada4e2c3d33f84821787cf0def66c: Status 404 returned error can't find the container with id 1d70d1c51b4b77bbd06f19f3c52015cbb2eada4e2c3d33f84821787cf0def66c
May 27 23:40:53.866300 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.866276    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff4dcda_d905_444a_88cc_49ab1ce8b7bf.slice/crio-5ac0b8e0ca53e560cefbf8a2cd6a3320a3d4ce052f93449e84a94b7d5e548898 WatchSource:0}: Error finding container 5ac0b8e0ca53e560cefbf8a2cd6a3320a3d4ce052f93449e84a94b7d5e548898: Status 404 returned error can't find the container with id 5ac0b8e0ca53e560cefbf8a2cd6a3320a3d4ce052f93449e84a94b7d5e548898
May 27 23:40:53.888245 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.888226    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0616e250_c245_4aa5_aaaa_ae15d9e3234e.slice/crio-77c0ec2e4c8888b558e8eb98424f1289ea4841f331f7b48e7a06d85a229857c3 WatchSource:0}: Error finding container 77c0ec2e4c8888b558e8eb98424f1289ea4841f331f7b48e7a06d85a229857c3: Status 404 returned error can't find the container with id 77c0ec2e4c8888b558e8eb98424f1289ea4841f331f7b48e7a06d85a229857c3
May 27 23:40:53.889506 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.889487    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe3b412_20be_435b_9aaa_ef0602a81cd5.slice/crio-86798aebdb82291602f04809d5c53975de0c92a13ba050dbab0137cdd399dd10 WatchSource:0}: Error finding container 86798aebdb82291602f04809d5c53975de0c92a13ba050dbab0137cdd399dd10: Status 404 returned error can't find the container with id 86798aebdb82291602f04809d5c53975de0c92a13ba050dbab0137cdd399dd10
May 27 23:40:53.890463 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.890415    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bba202_befe_4b9a_9ca9_88e58a64a498.slice/crio-71f07a7d3612da08490080fb77402d276afec6fb58a0d52e2bc3d607033c01d4 WatchSource:0}: Error finding container 71f07a7d3612da08490080fb77402d276afec6fb58a0d52e2bc3d607033c01d4: Status 404 returned error can't find the container with id 71f07a7d3612da08490080fb77402d276afec6fb58a0d52e2bc3d607033c01d4
May 27 23:40:53.891039 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.890936    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdb4626_9749_4c04_9159_2c63485fe159.slice/crio-3573618bf9a0a246d29df4959c09749c1f4a0970cd26d538cffe2eafd063637d WatchSource:0}: Error finding container 3573618bf9a0a246d29df4959c09749c1f4a0970cd26d538cffe2eafd063637d: Status 404 returned error can't find the container with id 3573618bf9a0a246d29df4959c09749c1f4a0970cd26d538cffe2eafd063637d
May 27 23:40:53.891809 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:40:53.891788    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d28d858_235d_4e96_9f9a_57634afaa5c9.slice/crio-01417554a3eda6aaba2bbdc94ac44cb723f1464b007a180d574dba790da34343 WatchSource:0}: Error finding container 01417554a3eda6aaba2bbdc94ac44cb723f1464b007a180d574dba790da34343: Status 404 returned error can't find the container with id 01417554a3eda6aaba2bbdc94ac44cb723f1464b007a180d574dba790da34343
May 27 23:40:54.177131 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.176965    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-26 23:35:52 +0000 UTC" deadline="2027-10-30 19:37:08.002049655 +0000 UTC"
May 27 23:40:54.177444 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.177131    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12499h56m13.824921643s"
May 27 23:40:54.221500 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.221468    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lw89q" event={"ID":"0616e250-c245-4aa5-aaaa-ae15d9e3234e","Type":"ContainerStarted","Data":"77c0ec2e4c8888b558e8eb98424f1289ea4841f331f7b48e7a06d85a229857c3"}
May 27 23:40:54.222230 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.222207    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n58tk" event={"ID":"0cdb4626-9749-4c04-9159-2c63485fe159","Type":"ContainerStarted","Data":"3573618bf9a0a246d29df4959c09749c1f4a0970cd26d538cffe2eafd063637d"}
May 27 23:40:54.222919 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.222897    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wvc2" event={"ID":"b2bba202-befe-4b9a-9ca9-88e58a64a498","Type":"ContainerStarted","Data":"71f07a7d3612da08490080fb77402d276afec6fb58a0d52e2bc3d607033c01d4"}
May 27 23:40:54.223586 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.223566    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"86798aebdb82291602f04809d5c53975de0c92a13ba050dbab0137cdd399dd10"}
May 27 23:40:54.224896 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.224875    2572 generic.go:358] "Generic (PLEG): container finished" podID="ae9a7b48520c58f68b30a66c3e150b8c" containerID="d75500dc19333d5a25d7b2e5f587e5b567d16feb950a29cf6e80c0727a882844" exitCode=0
May 27 23:40:54.224998 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.224931    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" event={"ID":"ae9a7b48520c58f68b30a66c3e150b8c","Type":"ContainerDied","Data":"d75500dc19333d5a25d7b2e5f587e5b567d16feb950a29cf6e80c0727a882844"}
May 27 23:40:54.226854 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.226813    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" event={"ID":"a329a27d84f7b98ef3b36fc9a735cfdf","Type":"ContainerStarted","Data":"bbeb27e879fe4e8c72c57c5c0cb748942532efcf52de6fb4a70f4b4b29a52a86"}
May 27 23:40:54.227787 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.227768    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hzvkg" event={"ID":"3d28d858-235d-4e96-9f9a-57634afaa5c9","Type":"ContainerStarted","Data":"01417554a3eda6aaba2bbdc94ac44cb723f1464b007a180d574dba790da34343"}
May 27 23:40:54.230984 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.229060    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerStarted","Data":"5ac0b8e0ca53e560cefbf8a2cd6a3320a3d4ce052f93449e84a94b7d5e548898"}
May 27 23:40:54.232135 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.232102    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flbbx" event={"ID":"8b3785e5-1e3a-4306-b7cb-64acde7e65ed","Type":"ContainerStarted","Data":"1d70d1c51b4b77bbd06f19f3c52015cbb2eada4e2c3d33f84821787cf0def66c"}
May 27 23:40:54.233189 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.233160    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7" event={"ID":"a6462972-dbcb-47e2-85b2-a43fac1480f8","Type":"ContainerStarted","Data":"f134e1895d59993f3f19d3d60e2a7147764a22576d6444ef2ed9315527bf7eb2"}
May 27 23:40:54.234068 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.234044    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pjwfd" event={"ID":"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c","Type":"ContainerStarted","Data":"441215dd6e12fb5eb70d75ca04df9cf50e3f7677177b6e61ac7ebd51819dddb7"}
May 27 23:40:54.839613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.839575    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:54.839805 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:54.839637    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:54.839805 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:54.839776    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:40:54.839805 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:54.839793    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:40:54.839805 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:54.839805    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:54.840003 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:54.839861    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:40:56.839843688 +0000 UTC m=+6.229347486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:54.840262 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:54.840244    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:54.840357 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:54.840299    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:40:56.84028383 +0000 UTC m=+6.229787637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:55.218229 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:55.218197    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:55.218672 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:55.218352    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:40:55.218672 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:55.218448    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:55.218672 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:55.218518    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:40:55.250743 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:55.250367    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" event={"ID":"ae9a7b48520c58f68b30a66c3e150b8c","Type":"ContainerStarted","Data":"54f313b22c618cc1f22075de455ae680a509088120b720498e2595bd4cd333c1"}
May 27 23:40:55.267873 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:55.267819    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" podStartSLOduration=3.267801961 podStartE2EDuration="3.267801961s" podCreationTimestamp="2026-05-27 23:40:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:40:54.249904875 +0000 UTC m=+3.639408690" watchObservedRunningTime="2026-05-27 23:40:55.267801961 +0000 UTC m=+4.657305776"
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:56.855556    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:56.855621    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:56.855776    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:56.855796    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:56.855810    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:56.855871    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:41:00.855852474 +0000 UTC m=+10.245356273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:56.856276    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:56.856388 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:56.856353    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:41:00.856329198 +0000 UTC m=+10.245832992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:40:57.215456 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.215420    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:57.215649 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:57.215558    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:40:57.216027 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.216009    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:57.216122 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:57.216102    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:40:57.852020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.851962    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" podStartSLOduration=5.851941012 podStartE2EDuration="5.851941012s" podCreationTimestamp="2026-05-27 23:40:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:40:55.268456554 +0000 UTC m=+4.657960369" watchObservedRunningTime="2026-05-27 23:40:57.851941012 +0000 UTC m=+7.241444846"
May 27 23:40:57.852903 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.852879    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rkp2l"]
May 27 23:40:57.856360 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.855973    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:57.856360 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:57.856052    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:40:57.964728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.963025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6554f633-c725-4f96-bb1c-f019f8185210-kubelet-config\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:57.964728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.963077    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:57.964728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:57.963157    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6554f633-c725-4f96-bb1c-f019f8185210-dbus\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.064202 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:58.064157    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6554f633-c725-4f96-bb1c-f019f8185210-dbus\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.064383 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:58.064229    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6554f633-c725-4f96-bb1c-f019f8185210-kubelet-config\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.064383 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:58.064259    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.064383 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:58.064365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6554f633-c725-4f96-bb1c-f019f8185210-dbus\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.064558 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:58.064395    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 23:40:58.064558 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:58.064433    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6554f633-c725-4f96-bb1c-f019f8185210-kubelet-config\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.064558 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:58.064474    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret podName:6554f633-c725-4f96-bb1c-f019f8185210 nodeName:}" failed. No retries permitted until 2026-05-27 23:40:58.56445367 +0000 UTC m=+7.953957462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret") pod "global-pull-secret-syncer-rkp2l" (UID: "6554f633-c725-4f96-bb1c-f019f8185210") : object "kube-system"/"original-pull-secret" not registered
May 27 23:40:58.569677 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:58.569044    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:58.569677 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:58.569229    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 23:40:58.569677 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:58.569297    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret podName:6554f633-c725-4f96-bb1c-f019f8185210 nodeName:}" failed. No retries permitted until 2026-05-27 23:40:59.569275395 +0000 UTC m=+8.958779194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret") pod "global-pull-secret-syncer-rkp2l" (UID: "6554f633-c725-4f96-bb1c-f019f8185210") : object "kube-system"/"original-pull-secret" not registered
May 27 23:40:59.217621 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:59.217590    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:40:59.218021 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:59.217715    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:40:59.218139 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:59.218120    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:40:59.218244 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:59.218226    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:40:59.218331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:59.218321    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:59.218441 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:59.218395    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:40:59.578864 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:40:59.578290    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:40:59.578864 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:59.578432    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 23:40:59.578864 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:40:59.578502    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret podName:6554f633-c725-4f96-bb1c-f019f8185210 nodeName:}" failed. No retries permitted until 2026-05-27 23:41:01.578482796 +0000 UTC m=+10.967986590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret") pod "global-pull-secret-syncer-rkp2l" (UID: "6554f633-c725-4f96-bb1c-f019f8185210") : object "kube-system"/"original-pull-secret" not registered
May 27 23:41:00.890132 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:00.890095    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:00.890199    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:00.890292    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:00.890299    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:00.890370    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:00.890373    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:41:08.890353089 +0000 UTC m=+18.279856881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:00.890387    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:41:00.890618 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:00.890516    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:41:08.890464379 +0000 UTC m=+18.279968193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:41:01.217057 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:01.217027    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:01.217223 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:01.217141    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:01.217537 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:01.217511    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:01.217633 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:01.217612    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:01.217676 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:01.217664    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:01.217775 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:01.217757    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:01.596712 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:01.596628    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:01.596864 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:01.596792    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 23:41:01.596864 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:01.596853    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret podName:6554f633-c725-4f96-bb1c-f019f8185210 nodeName:}" failed. No retries permitted until 2026-05-27 23:41:05.596839048 +0000 UTC m=+14.986342840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret") pod "global-pull-secret-syncer-rkp2l" (UID: "6554f633-c725-4f96-bb1c-f019f8185210") : object "kube-system"/"original-pull-secret" not registered
May 27 23:41:03.217501 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:03.217463    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:03.217501 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:03.217505    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:03.218038 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:03.217482    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:03.218038 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:03.217587    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:03.218038 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:03.217675    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:03.218038 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:03.217772    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:05.215561 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:05.215529    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:05.215989 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:05.215656    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:05.215989 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:05.215708    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:05.215989 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:05.215773    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:05.215989 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:05.215906    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:05.216145 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:05.216003    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:05.626932 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:05.626846    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:05.627078 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:05.627013    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 23:41:05.627143 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:05.627089    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret podName:6554f633-c725-4f96-bb1c-f019f8185210 nodeName:}" failed. No retries permitted until 2026-05-27 23:41:13.62707133 +0000 UTC m=+23.016575124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret") pod "global-pull-secret-syncer-rkp2l" (UID: "6554f633-c725-4f96-bb1c-f019f8185210") : object "kube-system"/"original-pull-secret" not registered
May 27 23:41:07.215696 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:07.215664    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:07.215696 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:07.215714    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:07.216176 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:07.215808    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:07.216176 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:07.215933    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:07.216176 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:07.216041    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:07.216176 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:07.216133    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:08.952427 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:08.952392    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:08.952479    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:08.952561    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:08.952588    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:08.952596    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:08.952660    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:41:24.952644903 +0000 UTC m=+34.342148699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:08.952601    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:41:08.952834 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:08.952740    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:41:24.952724073 +0000 UTC m=+34.342227884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:41:09.216150 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:09.216080    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:09.216332 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:09.216205    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:09.216332 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:09.216213    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:09.216437 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:09.216323    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:09.216437 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:09.216369    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:09.216539 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:09.216444    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:11.216451 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:11.216406    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:11.216451 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:11.216442    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:11.216881 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:11.216414    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:11.216881 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:11.216562    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:11.216881 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:11.216642    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:11.216881 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:11.216740    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:12.282847 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.282665    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"9c3347eba536e4b62353467f62b40cca4a52736b49cdf0babe611de4d3b479d9"}
May 27 23:41:12.284367 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.283898    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hzvkg" event={"ID":"3d28d858-235d-4e96-9f9a-57634afaa5c9","Type":"ContainerStarted","Data":"5767d17222a4d453596fbff18a7a864b956dbfbfda5d2330a314723624d8ca4a"}
May 27 23:41:12.286746 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.286710    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flbbx" event={"ID":"8b3785e5-1e3a-4306-b7cb-64acde7e65ed","Type":"ContainerStarted","Data":"a0fd18efb57ef51f4c61ad62ebd4ff76a58406cfa7bd544108c73f0ede0e2d98"}
May 27 23:41:12.289256 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.289232    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lw89q" event={"ID":"0616e250-c245-4aa5-aaaa-ae15d9e3234e","Type":"ContainerStarted","Data":"1a64b1dd5439c1e549086c62dfffcf5f2296c821b8474e9f6e373f803d509bd4"}
May 27 23:41:12.292151 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.291626    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wvc2" event={"ID":"b2bba202-befe-4b9a-9ca9-88e58a64a498","Type":"ContainerStarted","Data":"834f770f8c9f772e795e347f9ef4cf13c7f048050dcd13679ad5cdd64578ee4c"}
May 27 23:41:12.307022 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.306582    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hzvkg" podStartSLOduration=6.953528566 podStartE2EDuration="21.30657246s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.895080716 +0000 UTC m=+3.284584508" lastFinishedPulling="2026-05-27 23:41:08.248124611 +0000 UTC m=+17.637628402" observedRunningTime="2026-05-27 23:41:12.306278511 +0000 UTC m=+21.695782326" watchObservedRunningTime="2026-05-27 23:41:12.30657246 +0000 UTC m=+21.696076293"
May 27 23:41:12.322704 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.322639    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-flbbx" podStartSLOduration=6.939176997 podStartE2EDuration="21.32262239s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.864682988 +0000 UTC m=+3.254186781" lastFinishedPulling="2026-05-27 23:41:08.248128382 +0000 UTC m=+17.637632174" observedRunningTime="2026-05-27 23:41:12.32237674 +0000 UTC m=+21.711880555" watchObservedRunningTime="2026-05-27 23:41:12.32262239 +0000 UTC m=+21.712126207"
May 27 23:41:12.342220 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:12.342140    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wvc2" podStartSLOduration=3.35402739 podStartE2EDuration="21.342032195s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.895125994 +0000 UTC m=+3.284629787" lastFinishedPulling="2026-05-27 23:41:11.883130786 +0000 UTC m=+21.272634592" observedRunningTime="2026-05-27 23:41:12.341610597 +0000 UTC m=+21.731114409" watchObservedRunningTime="2026-05-27 23:41:12.342032195 +0000 UTC m=+21.731535988"
May 27 23:41:13.215876 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.215648    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:13.216097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.215675    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:13.216097 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:13.215907    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:13.216097 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:13.216005    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:13.216097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.215704    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:13.216261 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:13.216121    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:13.295474 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295447    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:41:13.296127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295742    2572 generic.go:358] "Generic (PLEG): container finished" podID="6fe3b412-20be-435b-9aaa-ef0602a81cd5" containerID="d9b56d549e348caf9a64f73d010645112bb556cbc6f60a49d71c9beb9bebb56e" exitCode=1
May 27 23:41:13.296127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295800    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"680349a1edbebacbb946f74d4727820f549dd1c79c068908f5b4f94aa929deb5"}
May 27 23:41:13.296127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295833    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"c138a6adf30adcbe701f6e5884882914b6eab38e58b4ca2d074b63e7cea0bde1"}
May 27 23:41:13.296127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295843    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"c92222accaa5b7b9afca8b1c32a2def032f79a0116271b0d649df00e9cb0a670"}
May 27 23:41:13.296127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295852    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"b4190111d15671eff6adda8dd3a5454f0521dd9d75004557d1e988369df36486"}
May 27 23:41:13.296127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.295860    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerDied","Data":"d9b56d549e348caf9a64f73d010645112bb556cbc6f60a49d71c9beb9bebb56e"}
May 27 23:41:13.297095 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.297073    2572 generic.go:358] "Generic (PLEG): container finished" podID="8ff4dcda-d905-444a-88cc-49ab1ce8b7bf" containerID="53629e84ad9625eed1828b463a590e109b59f7310580839c0b7610c96cf59422" exitCode=0
May 27 23:41:13.297193 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.297132    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerDied","Data":"53629e84ad9625eed1828b463a590e109b59f7310580839c0b7610c96cf59422"}
May 27 23:41:13.298446 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.298423    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7" event={"ID":"a6462972-dbcb-47e2-85b2-a43fac1480f8","Type":"ContainerStarted","Data":"bb43b1e29b3a1b9059cd04d5570df635cc38f0c1b02262e4df14635c99e31727"}
May 27 23:41:13.299614 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.299589    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pjwfd" event={"ID":"69ebaa5a-f129-4be0-b4a7-3dc11eafd58c","Type":"ContainerStarted","Data":"95ab5d5323439b217a05382954fc2e78b1fe8a018c6fd0cbe2e5b2c0baa68824"}
May 27 23:41:13.337713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.337665    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lw89q" podStartSLOduration=4.3707246 podStartE2EDuration="22.337649237s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.889905337 +0000 UTC m=+3.279409136" lastFinishedPulling="2026-05-27 23:41:11.856829978 +0000 UTC m=+21.246333773" observedRunningTime="2026-05-27 23:41:13.337585843 +0000 UTC m=+22.727089657" watchObservedRunningTime="2026-05-27 23:41:13.337649237 +0000 UTC m=+22.727153051"
May 27 23:41:13.358174 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.358128    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-pjwfd" podStartSLOduration=3.9897383 podStartE2EDuration="22.35811681s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.860176838 +0000 UTC m=+3.249680630" lastFinishedPulling="2026-05-27 23:41:12.228555335 +0000 UTC m=+21.618059140" observedRunningTime="2026-05-27 23:41:13.358078889 +0000 UTC m=+22.747582702" watchObservedRunningTime="2026-05-27 23:41:13.35811681 +0000 UTC m=+22.747620625"
May 27 23:41:13.493626 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.493604    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 27 23:41:13.687292 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:13.687265    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:13.687453 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:13.687412    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 23:41:13.687506 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:13.687467    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret podName:6554f633-c725-4f96-bb1c-f019f8185210 nodeName:}" failed. No retries permitted until 2026-05-27 23:41:29.687451839 +0000 UTC m=+39.076955631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret") pod "global-pull-secret-syncer-rkp2l" (UID: "6554f633-c725-4f96-bb1c-f019f8185210") : object "kube-system"/"original-pull-secret" not registered
May 27 23:41:14.196984 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:14.196877    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-27T23:41:13.493620809Z","UUID":"465fe96a-7847-49f3-b3c0-0958e879da7d","Handler":null,"Name":"","Endpoint":""}
May 27 23:41:14.199495 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:14.199215    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 27 23:41:14.199495 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:14.199243    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 27 23:41:14.302687 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:14.302658    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7" event={"ID":"a6462972-dbcb-47e2-85b2-a43fac1480f8","Type":"ContainerStarted","Data":"5d14980aa38b202798ab7bc5736a0cf22ac0e2b60fd94a1d524907d393b76b31"}
May 27 23:41:14.303775 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:14.303746    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n58tk" event={"ID":"0cdb4626-9749-4c04-9159-2c63485fe159","Type":"ContainerStarted","Data":"2f511a3713b512bd0da5d2c60cfd40289883a25bb11e66fce4f25dedf3dc9198"}
May 27 23:41:14.320463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:14.320417    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n58tk" podStartSLOduration=5.328223529 podStartE2EDuration="23.320400763s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.895102751 +0000 UTC m=+3.284606544" lastFinishedPulling="2026-05-27 23:41:11.887279987 +0000 UTC m=+21.276783778" observedRunningTime="2026-05-27 23:41:14.319842754 +0000 UTC m=+23.709346567" watchObservedRunningTime="2026-05-27 23:41:14.320400763 +0000 UTC m=+23.709904579"
May 27 23:41:15.215892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.215667    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:15.216027 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.215674    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:15.216027 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.215684    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:15.216027 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:15.216012    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:15.216149 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:15.216102    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:15.216189 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:15.216175    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:15.309070 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.308998    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:41:15.309515 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.309383    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"01811a12440bd044d924d7a0acc4452cf29f5a1e489bee6a85eb8a571135aca9"}
May 27 23:41:15.311530 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.311502    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7" event={"ID":"a6462972-dbcb-47e2-85b2-a43fac1480f8","Type":"ContainerStarted","Data":"4855f7dd88c2037fca61b181def388efa54ca9d3eb0c1603e97f80995f805595"}
May 27 23:41:15.331593 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.331551    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjwj7" podStartSLOduration=3.819984687 podStartE2EDuration="24.33153478s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.863616005 +0000 UTC m=+3.253119806" lastFinishedPulling="2026-05-27 23:41:14.3751661 +0000 UTC m=+23.764669899" observedRunningTime="2026-05-27 23:41:15.331509521 +0000 UTC m=+24.721013337" watchObservedRunningTime="2026-05-27 23:41:15.33153478 +0000 UTC m=+24.721038595"
May 27 23:41:15.740773 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.740739    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:41:15.741295 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:15.741279    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:41:16.768075 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:16.768036    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:41:16.768633 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:16.768572    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hzvkg"
May 27 23:41:17.215112 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:17.215074    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:17.215112 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:17.215100    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:17.215332 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:17.215093    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:17.215332 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:17.215188    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:17.215332 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:17.215248    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:17.215492 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:17.215335    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:19.216237 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.216061    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:19.216856 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.216075    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:19.216856 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:19.216357    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:19.216856 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.216076    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:19.216856 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:19.216438    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:19.216856 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:19.216496    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:19.319793 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.319769    2572 generic.go:358] "Generic (PLEG): container finished" podID="8ff4dcda-d905-444a-88cc-49ab1ce8b7bf" containerID="6421e3d2caf09842e69a2c1cf4e640b791b38df2797ef2ad1d4c4d3904c066bf" exitCode=0
May 27 23:41:19.319931 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.319846    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerDied","Data":"6421e3d2caf09842e69a2c1cf4e640b791b38df2797ef2ad1d4c4d3904c066bf"}
May 27 23:41:19.322875 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.322860    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:41:19.323228 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.323198    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"e49366c21e8c35fa7b7d62cba9e856224b3ae8b1454a89643efd780a213ef354"}
May 27 23:41:19.323515 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.323502    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:41:19.323566 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.323524    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:41:19.323566 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.323537    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:41:19.323657 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.323640    2572 scope.go:117] "RemoveContainer" containerID="d9b56d549e348caf9a64f73d010645112bb556cbc6f60a49d71c9beb9bebb56e"
May 27 23:41:19.340117 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.340095    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:41:19.340203 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:19.340159    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:41:20.328554 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.328532    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:41:20.328868 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.328780    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" event={"ID":"6fe3b412-20be-435b-9aaa-ef0602a81cd5","Type":"ContainerStarted","Data":"34c47d742bb9d7e4f047f5ab00bf43628ba329fab5f41fb03c8ea5f256092d55"}
May 27 23:41:20.363735 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.363692    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv" podStartSLOduration=11.356163191 podStartE2EDuration="29.363678301s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.894986339 +0000 UTC m=+3.284490146" lastFinishedPulling="2026-05-27 23:41:11.902501463 +0000 UTC m=+21.292005256" observedRunningTime="2026-05-27 23:41:20.362429017 +0000 UTC m=+29.751932831" watchObservedRunningTime="2026-05-27 23:41:20.363678301 +0000 UTC m=+29.753182114"
May 27 23:41:20.449112 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.448990    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttrwc"]
May 27 23:41:20.449237 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.449146    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:20.449279 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:20.449231    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:20.449687 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.449665    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bp6lz"]
May 27 23:41:20.449790 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.449777    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:20.449906 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:20.449887    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:20.450281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.450261    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rkp2l"]
May 27 23:41:20.450418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:20.450403    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:20.450501 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:20.450480    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:21.332768 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:21.332728    2572 generic.go:358] "Generic (PLEG): container finished" podID="8ff4dcda-d905-444a-88cc-49ab1ce8b7bf" containerID="bff3b3ed0470a1738f77f6d47c2588c4dbdb319362ee7a406da67463e7331f10" exitCode=0
May 27 23:41:21.333204 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:21.332818    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerDied","Data":"bff3b3ed0470a1738f77f6d47c2588c4dbdb319362ee7a406da67463e7331f10"}
May 27 23:41:22.215826 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:22.215798    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:22.216005 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:22.215801    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:22.216065 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:22.216005    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:22.216065 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:22.215801    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:22.216141 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:22.215896    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:22.216141 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:22.216106    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:23.338410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:23.338301    2572 generic.go:358] "Generic (PLEG): container finished" podID="8ff4dcda-d905-444a-88cc-49ab1ce8b7bf" containerID="eb8778f64ed2bee81ecbdb1682e9ca0c15a64c8980ddc923c45fe90a41a04bb9" exitCode=0
May 27 23:41:23.338410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:23.338373    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerDied","Data":"eb8778f64ed2bee81ecbdb1682e9ca0c15a64c8980ddc923c45fe90a41a04bb9"}
May 27 23:41:24.215276 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.215242    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:24.215549 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.215242    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:24.215549 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.215365    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:41:24.215549 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.215243    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:24.215549 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.215426    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttrwc" podUID="cd519bd5-eacb-4a30-a0ec-6e554d8001da"
May 27 23:41:24.215549 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.215492    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkp2l" podUID="6554f633-c725-4f96-bb1c-f019f8185210"
May 27 23:41:24.922250 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.922216    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeReady"
May 27 23:41:24.922799 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.922405    2572 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 27 23:41:24.955530 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.955502    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d6847498c-gkxfs"]
May 27 23:41:24.973264 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.973234    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:24.973457 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.973287    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:24.973519 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.973479    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 23:41:24.973519 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.973494    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 23:41:24.973519 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.973502    2572 projected.go:194] Error preparing data for projected volume kube-api-access-f7jz5 for pod openshift-network-diagnostics/network-check-target-ttrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:41:24.973624 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.973544    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5 podName:cd519bd5-eacb-4a30-a0ec-6e554d8001da nodeName:}" failed. No retries permitted until 2026-05-27 23:41:56.973532564 +0000 UTC m=+66.363036356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7jz5" (UniqueName: "kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5") pod "network-check-target-ttrwc" (UID: "cd519bd5-eacb-4a30-a0ec-6e554d8001da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 23:41:24.973868 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.973853    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:41:24.973946 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:24.973905    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:41:56.973886564 +0000 UTC m=+66.363390355 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 23:41:24.975732 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.975701    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-44grr"]
May 27 23:41:24.979398 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.975960    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:24.981433 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.979641    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rf4mt\""
May 27 23:41:24.981433 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.979692    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 27 23:41:24.981433 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.979925    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 27 23:41:24.981433 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.980063    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 27 23:41:24.985919 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.985896    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 27 23:41:24.997018 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.996999    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-22265"]
May 27 23:41:24.997186 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.997171    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44grr"
May 27 23:41:24.999556 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.999534    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 27 23:41:24.999655 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.999575    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 27 23:41:24.999655 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:24.999575    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mpjmq\""
May 27 23:41:25.015004 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.014983    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6847498c-gkxfs"]
May 27 23:41:25.015004 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.015005    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-22265"]
May 27 23:41:25.015130 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.015012    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-44grr"]
May 27 23:41:25.015130 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.015100    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.026443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.026422    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 27 23:41:25.026547 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.026532    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2gd8z\""
May 27 23:41:25.026788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.026769    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 27 23:41:25.028185 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.028169    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 27 23:41:25.074419 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.074575 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074428    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkxn\" (UniqueName: \"kubernetes.io/projected/b39519dd-a77f-4429-913c-18762f94546d-kube-api-access-vdkxn\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.074575 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074464    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-image-registry-private-configuration\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074575 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39519dd-a77f-4429-913c-18762f94546d-config-volume\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.074575 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-certificates\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074602    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-ca-trust-extracted\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-trusted-ca\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-bound-sa-token\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074707    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-installation-pull-secrets\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75f8h\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-kube-api-access-75f8h\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.074970 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.074771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39519dd-a77f-4429-913c-18762f94546d-tmp-dir\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175113 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175044    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-trusted-ca\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175113 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175074    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-bound-sa-token\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175113 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175100    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-installation-pull-secrets\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175118    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-75f8h\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-kube-api-access-75f8h\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175137    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39519dd-a77f-4429-913c-18762f94546d-tmp-dir\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175157    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175177    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkxn\" (UniqueName: \"kubernetes.io/projected/b39519dd-a77f-4429-913c-18762f94546d-kube-api-access-vdkxn\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99g8\" (UniqueName: \"kubernetes.io/projected/30d6a16f-881b-4b68-892c-e306fd38659b-kube-api-access-z99g8\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175241    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-image-registry-private-configuration\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175265    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39519dd-a77f-4429-913c-18762f94546d-config-volume\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.175271    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175299    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-certificates\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175418 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.175352    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:41:25.675331861 +0000 UTC m=+35.064835668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175517    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39519dd-a77f-4429-913c-18762f94546d-tmp-dir\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175622    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175648    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-ca-trust-extracted\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.175681    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.175698    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.175742    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:41:25.675725845 +0000 UTC m=+35.065229638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175851    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39519dd-a77f-4429-913c-18762f94546d-config-volume\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.175944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.175937    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-ca-trust-extracted\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.176279 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.176053    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-certificates\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.176279 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.176231    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-trusted-ca\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.179422 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.179402    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-image-registry-private-configuration\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.179495 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.179404    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-installation-pull-secrets\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.183986 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.183960    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-bound-sa-token\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.184465 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.184449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkxn\" (UniqueName: \"kubernetes.io/projected/b39519dd-a77f-4429-913c-18762f94546d-kube-api-access-vdkxn\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.184556 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.184519    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f8h\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-kube-api-access-75f8h\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.276669 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.276625    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z99g8\" (UniqueName: \"kubernetes.io/projected/30d6a16f-881b-4b68-892c-e306fd38659b-kube-api-access-z99g8\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.276818 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.276694    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.276882 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.276833    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:25.276952 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.276912    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:41:25.776890642 +0000 UTC m=+35.166394441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:25.287665 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.287639    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99g8\" (UniqueName: \"kubernetes.io/projected/30d6a16f-881b-4b68-892c-e306fd38659b-kube-api-access-z99g8\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.681046 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.681012    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:25.681208 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.681084    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:25.681208 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.681187    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:25.681299 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.681211    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:25.681299 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.681194    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:25.681299 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.681283    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:41:26.681262844 +0000 UTC m=+36.070766641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:25.681456 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.681348    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:41:26.681329835 +0000 UTC m=+36.070833643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:25.782114 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:25.782077    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:25.782266 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.782233    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:25.782344 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:25.782322    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:41:26.782286831 +0000 UTC m=+36.171790638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:26.216132 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.216098    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:26.216683 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.216103    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:26.216683 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.216115    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:26.219673 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.219651    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 27 23:41:26.219782 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.219704    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nr2\""
May 27 23:41:26.219782 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.219742    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 27 23:41:26.219782 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.219765    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 27 23:41:26.219782 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.219771    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qfthh\""
May 27 23:41:26.220007 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.219992    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 27 23:41:26.687653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.687615    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:26.687839 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.687678    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:26.687839 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.687779    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:26.687839 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.687790    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:26.687839 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.687790    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:26.687839 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.687841    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:41:28.687827586 +0000 UTC m=+38.077331378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:26.688030 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.687853    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:41:28.687847145 +0000 UTC m=+38.077350937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:26.789094 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:26.788993    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:26.789252 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.789159    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:26.789252 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:26.789232    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:41:28.789212741 +0000 UTC m=+38.178716532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:28.704453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:28.704405    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:28.704936 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:28.704517    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:28.704936 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.704567    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:28.704936 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.704638    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:41:32.704619951 +0000 UTC m=+42.094123745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:28.704936 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.704664    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:28.704936 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.704687    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:28.704936 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.704743    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:41:32.704724333 +0000 UTC m=+42.094228142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:28.805809 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:28.805751    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:28.805989 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.805923    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:28.806055 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:28.806001    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:41:32.805980673 +0000 UTC m=+42.195484464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:29.713887 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:29.713706    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:29.716748 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:29.716727    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6554f633-c725-4f96-bb1c-f019f8185210-original-pull-secret\") pod \"global-pull-secret-syncer-rkp2l\" (UID: \"6554f633-c725-4f96-bb1c-f019f8185210\") " pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:29.838942 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:29.838899    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkp2l"
May 27 23:41:29.976102 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:29.976030    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rkp2l"]
May 27 23:41:29.979021 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:41:29.978998    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6554f633_c725_4f96_bb1c_f019f8185210.slice/crio-8ec073fc9d650dbdc0ce877f8c00bb79308f9d5960034b472eceef73a0815c36 WatchSource:0}: Error finding container 8ec073fc9d650dbdc0ce877f8c00bb79308f9d5960034b472eceef73a0815c36: Status 404 returned error can't find the container with id 8ec073fc9d650dbdc0ce877f8c00bb79308f9d5960034b472eceef73a0815c36
May 27 23:41:30.353560 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:30.353481    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rkp2l" event={"ID":"6554f633-c725-4f96-bb1c-f019f8185210","Type":"ContainerStarted","Data":"8ec073fc9d650dbdc0ce877f8c00bb79308f9d5960034b472eceef73a0815c36"}
May 27 23:41:30.355768 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:30.355745    2572 generic.go:358] "Generic (PLEG): container finished" podID="8ff4dcda-d905-444a-88cc-49ab1ce8b7bf" containerID="0b766497b79865d673a093ec8ca0afaf0e2c27efd035349d73414f9e75ac50a4" exitCode=0
May 27 23:41:30.355834 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:30.355778    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerDied","Data":"0b766497b79865d673a093ec8ca0afaf0e2c27efd035349d73414f9e75ac50a4"}
May 27 23:41:31.360260 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:31.360219    2572 generic.go:358] "Generic (PLEG): container finished" podID="8ff4dcda-d905-444a-88cc-49ab1ce8b7bf" containerID="493a4b264ada1f5368a042c047114ba268059c47c6aba513ad3d5887b2cf0a28" exitCode=0
May 27 23:41:31.360653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:31.360287    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerDied","Data":"493a4b264ada1f5368a042c047114ba268059c47c6aba513ad3d5887b2cf0a28"}
May 27 23:41:32.365799 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:32.365762    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" event={"ID":"8ff4dcda-d905-444a-88cc-49ab1ce8b7bf","Type":"ContainerStarted","Data":"9dcf1f6063b3f6b7304163c97a0a52403c043c86a61fb9c9eb46596015f9191c"}
May 27 23:41:32.404492 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:32.404427    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rzrfc" podStartSLOduration=5.993623524 podStartE2EDuration="41.404408179s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:40:53.887230311 +0000 UTC m=+3.276734103" lastFinishedPulling="2026-05-27 23:41:29.298014955 +0000 UTC m=+38.687518758" observedRunningTime="2026-05-27 23:41:32.400862082 +0000 UTC m=+41.790365897" watchObservedRunningTime="2026-05-27 23:41:32.404408179 +0000 UTC m=+41.793911995"
May 27 23:41:32.736633 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:32.736595    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:32.736814 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:32.736655    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:32.736814 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.736757    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:32.736814 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.736788    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:32.736814 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.736799    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:32.736992 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.736828    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:41:40.736810609 +0000 UTC m=+50.126314414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:32.736992 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.736844    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:41:40.736837359 +0000 UTC m=+50.126341151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:32.837637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:32.837609    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:32.837841 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.837766    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:32.837887 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:32.837842    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:41:40.837821638 +0000 UTC m=+50.227325435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:35.373145 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:35.373110    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rkp2l" event={"ID":"6554f633-c725-4f96-bb1c-f019f8185210","Type":"ContainerStarted","Data":"8cf7d9bc48c191b9f852265574eac107d68a27bc51951cf9347549d858f8cbb7"}
May 27 23:41:35.388038 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:35.387986    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rkp2l" podStartSLOduration=33.914274586 podStartE2EDuration="38.387970201s" podCreationTimestamp="2026-05-27 23:40:57 +0000 UTC" firstStartedPulling="2026-05-27 23:41:29.9806927 +0000 UTC m=+39.370196493" lastFinishedPulling="2026-05-27 23:41:34.454388317 +0000 UTC m=+43.843892108" observedRunningTime="2026-05-27 23:41:35.387688162 +0000 UTC m=+44.777191977" watchObservedRunningTime="2026-05-27 23:41:35.387970201 +0000 UTC m=+44.777474016"
May 27 23:41:40.797821 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:40.797774    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:40.798360 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:40.797874    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:40.798360 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.797922    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:40.798360 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.797942    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:40.798360 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.797997    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:41:56.797982341 +0000 UTC m=+66.187486133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:40.798360 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.798008    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:40.798360 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.798082    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:41:56.798051087 +0000 UTC m=+66.187554884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:40.899126 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:40.899093    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:40.899262 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.899233    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:40.899321 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:40.899290    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:41:56.899273536 +0000 UTC m=+66.288777331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:51.343640 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:51.343613    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cpqgv"
May 27 23:41:56.806782 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:56.806725    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:41:56.807152 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:56.806818    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:41:56.807152 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.806892    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:41:56.807152 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.806921    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:41:56.807152 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.806932    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:41:56.807152 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.806965    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:42:28.806946056 +0000 UTC m=+98.196449869 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:41:56.807152 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.806983    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:42:28.806974595 +0000 UTC m=+98.196478389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:41:56.907782 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:56.907753    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:41:56.907906 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.907888    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:41:56.907957 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:56.907947    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:42:28.90793218 +0000 UTC m=+98.297436011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:41:57.008939 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.008909    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:41:57.009050 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.008949    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:57.011873 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.011855    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 27 23:41:57.011933 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.011890    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 27 23:41:57.019946 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:57.019926    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 27 23:41:57.020004 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:41:57.019988    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:43:01.019969992 +0000 UTC m=+130.409473797 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : secret "metrics-daemon-secret" not found
May 27 23:41:57.022421 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.022407    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 27 23:41:57.032928 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.032909    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jz5\" (UniqueName: \"kubernetes.io/projected/cd519bd5-eacb-4a30-a0ec-6e554d8001da-kube-api-access-f7jz5\") pod \"network-check-target-ttrwc\" (UID: \"cd519bd5-eacb-4a30-a0ec-6e554d8001da\") " pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:57.128701 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.128638    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qfthh\""
May 27 23:41:57.136887 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.136870    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:41:57.259632 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.259604    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttrwc"]
May 27 23:41:57.262859 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:41:57.262834    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd519bd5_eacb_4a30_a0ec_6e554d8001da.slice/crio-d2c190861751dac8478371dd7d71fe5c26529b0775d17925eb21297d18f1b134 WatchSource:0}: Error finding container d2c190861751dac8478371dd7d71fe5c26529b0775d17925eb21297d18f1b134: Status 404 returned error can't find the container with id d2c190861751dac8478371dd7d71fe5c26529b0775d17925eb21297d18f1b134
May 27 23:41:57.412961 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:41:57.412928    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttrwc" event={"ID":"cd519bd5-eacb-4a30-a0ec-6e554d8001da","Type":"ContainerStarted","Data":"d2c190861751dac8478371dd7d71fe5c26529b0775d17925eb21297d18f1b134"}
May 27 23:42:00.420506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:00.420468    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttrwc" event={"ID":"cd519bd5-eacb-4a30-a0ec-6e554d8001da","Type":"ContainerStarted","Data":"ea7c1039f113b54fd8e5339d9ea1d86be6bff1d38fd922bc01a6a5c68600e0ea"}
May 27 23:42:00.420849 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:00.420606    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:42:00.436771 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:00.436723    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ttrwc" podStartSLOduration=66.457665221 podStartE2EDuration="1m9.436710529s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:41:57.264649219 +0000 UTC m=+66.654153012" lastFinishedPulling="2026-05-27 23:42:00.243694529 +0000 UTC m=+69.633198320" observedRunningTime="2026-05-27 23:42:00.436416149 +0000 UTC m=+69.825919964" watchObservedRunningTime="2026-05-27 23:42:00.436710529 +0000 UTC m=+69.826214343"
May 27 23:42:28.835037 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:28.834900    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:42:28.835037 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:28.834970    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:42:28.835548 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.835068    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 23:42:28.835548 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.835068    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 23:42:28.835548 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.835135    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls podName:b39519dd-a77f-4429-913c-18762f94546d nodeName:}" failed. No retries permitted until 2026-05-27 23:43:32.835118875 +0000 UTC m=+162.224622667 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls") pod "dns-default-44grr" (UID: "b39519dd-a77f-4429-913c-18762f94546d") : secret "dns-default-metrics-tls" not found
May 27 23:42:28.835548 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.835079    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6847498c-gkxfs: secret "image-registry-tls" not found
May 27 23:42:28.835548 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.835193    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls podName:fe0b8634-6037-4df8-9cb0-a35ff9b57dff nodeName:}" failed. No retries permitted until 2026-05-27 23:43:32.835182136 +0000 UTC m=+162.224685929 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls") pod "image-registry-6d6847498c-gkxfs" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff") : secret "image-registry-tls" not found
May 27 23:42:28.935590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:28.935548    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:42:28.935730 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.935686    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 23:42:28.935771 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:42:28.935735    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert podName:30d6a16f-881b-4b68-892c-e306fd38659b nodeName:}" failed. No retries permitted until 2026-05-27 23:43:32.935722886 +0000 UTC m=+162.325226678 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert") pod "ingress-canary-22265" (UID: "30d6a16f-881b-4b68-892c-e306fd38659b") : secret "canary-serving-cert" not found
May 27 23:42:31.424213 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:42:31.424181    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ttrwc"
May 27 23:43:01.046079 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:01.046038    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:43:01.046602 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:01.046181    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 27 23:43:01.046602 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:01.046246    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs podName:80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f nodeName:}" failed. No retries permitted until 2026-05-27 23:45:03.046230991 +0000 UTC m=+252.435734783 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs") pod "network-metrics-daemon-bp6lz" (UID: "80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f") : secret "metrics-daemon-secret" not found
May 27 23:43:12.176845 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:12.176820    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flbbx_8b3785e5-1e3a-4306-b7cb-64acde7e65ed/dns-node-resolver/0.log"
May 27 23:43:13.176463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:13.176433    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lw89q_0616e250-c245-4aa5-aaaa-ae15d9e3234e/node-ca/0.log"
May 27 23:43:14.149612 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.149583    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"]
May 27 23:43:14.152532 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.152515    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.154806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.154784    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 27 23:43:14.154806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.154795    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 27 23:43:14.155127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.155112    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lcb42\""
May 27 23:43:14.155792 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.155745    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 27 23:43:14.160524 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.160462    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"]
May 27 23:43:14.244133 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.244099    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.244133 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.244133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zng9\" (UniqueName: \"kubernetes.io/projected/fddfa466-8322-4bb5-a94f-ad100bb45c81-kube-api-access-9zng9\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.344723 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.344690    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.344850 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.344728    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9zng9\" (UniqueName: \"kubernetes.io/projected/fddfa466-8322-4bb5-a94f-ad100bb45c81-kube-api-access-9zng9\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.344850 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:14.344844    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:43:14.345020 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:14.344917    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls podName:fddfa466-8322-4bb5-a94f-ad100bb45c81 nodeName:}" failed. No retries permitted until 2026-05-27 23:43:14.844899783 +0000 UTC m=+144.234403594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls") pod "cluster-samples-operator-697587599-64b5x" (UID: "fddfa466-8322-4bb5-a94f-ad100bb45c81") : secret "samples-operator-tls" not found
May 27 23:43:14.354170 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.354140    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zng9\" (UniqueName: \"kubernetes.io/projected/fddfa466-8322-4bb5-a94f-ad100bb45c81-kube-api-access-9zng9\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.849520 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:14.849482    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:14.849688 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:14.849628    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:43:14.849728 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:14.849697    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls podName:fddfa466-8322-4bb5-a94f-ad100bb45c81 nodeName:}" failed. No retries permitted until 2026-05-27 23:43:15.849678801 +0000 UTC m=+145.239182592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls") pod "cluster-samples-operator-697587599-64b5x" (UID: "fddfa466-8322-4bb5-a94f-ad100bb45c81") : secret "samples-operator-tls" not found
May 27 23:43:15.856067 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:15.856029    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:15.856577 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:15.856194    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:43:15.856577 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:15.856277    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls podName:fddfa466-8322-4bb5-a94f-ad100bb45c81 nodeName:}" failed. No retries permitted until 2026-05-27 23:43:17.856255571 +0000 UTC m=+147.245759363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls") pod "cluster-samples-operator-697587599-64b5x" (UID: "fddfa466-8322-4bb5-a94f-ad100bb45c81") : secret "samples-operator-tls" not found
May 27 23:43:16.321873 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.321833    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"]
May 27 23:43:16.324639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.324622    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.326988 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.326963    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 27 23:43:16.327103 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.326986    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 27 23:43:16.327103 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.326965    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 27 23:43:16.327238 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.327221    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 27 23:43:16.327863 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.327847    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kw7cp\""
May 27 23:43:16.331994 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.331975    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"]
May 27 23:43:16.434754 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.434722    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"]
May 27 23:43:16.437474 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.437455    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.439822 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.439802    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 27 23:43:16.439914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.439829    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 27 23:43:16.439914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.439807    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 27 23:43:16.439914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.439854    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 27 23:43:16.440067 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.439932    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rzc58\""
May 27 23:43:16.444477 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.444461    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"]
May 27 23:43:16.460356 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.460335    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fk79\" (UniqueName: \"kubernetes.io/projected/4717a5f4-dc35-49e8-a0f1-b2e008417733-kube-api-access-2fk79\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.460468 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.460369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4717a5f4-dc35-49e8-a0f1-b2e008417733-config\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.460468 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.460390    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4717a5f4-dc35-49e8-a0f1-b2e008417733-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.560847 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.560818    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2fk79\" (UniqueName: \"kubernetes.io/projected/4717a5f4-dc35-49e8-a0f1-b2e008417733-kube-api-access-2fk79\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.560847 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.560849    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4717a5f4-dc35-49e8-a0f1-b2e008417733-config\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.561060 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.560877    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4717a5f4-dc35-49e8-a0f1-b2e008417733-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.561060 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.561028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5194cbd-67e6-43a3-8698-91ce60449345-config\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.561166 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.561074    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtsp\" (UniqueName: \"kubernetes.io/projected/d5194cbd-67e6-43a3-8698-91ce60449345-kube-api-access-rmtsp\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.561166 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.561139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5194cbd-67e6-43a3-8698-91ce60449345-serving-cert\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.561405 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.561386    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4717a5f4-dc35-49e8-a0f1-b2e008417733-config\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.563066 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.563045    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4717a5f4-dc35-49e8-a0f1-b2e008417733-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.569478 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.569459    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fk79\" (UniqueName: \"kubernetes.io/projected/4717a5f4-dc35-49e8-a0f1-b2e008417733-kube-api-access-2fk79\") pod \"kube-storage-version-migrator-operator-565644f49-ddqvm\" (UID: \"4717a5f4-dc35-49e8-a0f1-b2e008417733\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.634172 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.634109    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"
May 27 23:43:16.661988 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.661953    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5194cbd-67e6-43a3-8698-91ce60449345-serving-cert\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.662110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.662031    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5194cbd-67e6-43a3-8698-91ce60449345-config\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.662110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.662072    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtsp\" (UniqueName: \"kubernetes.io/projected/d5194cbd-67e6-43a3-8698-91ce60449345-kube-api-access-rmtsp\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.662546 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.662523    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5194cbd-67e6-43a3-8698-91ce60449345-config\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.664229 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.664211    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5194cbd-67e6-43a3-8698-91ce60449345-serving-cert\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.670852 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.670830    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtsp\" (UniqueName: \"kubernetes.io/projected/d5194cbd-67e6-43a3-8698-91ce60449345-kube-api-access-rmtsp\") pod \"service-ca-operator-57b5b68b8-fhtrw\" (UID: \"d5194cbd-67e6-43a3-8698-91ce60449345\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.746637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.746600    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"
May 27 23:43:16.755086 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.755059    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm"]
May 27 23:43:16.757854 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:16.757831    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4717a5f4_dc35_49e8_a0f1_b2e008417733.slice/crio-357226bf45d1a5aaf86ef15a48abc03e77f9b596fa4f43f44043330fe06f30cd WatchSource:0}: Error finding container 357226bf45d1a5aaf86ef15a48abc03e77f9b596fa4f43f44043330fe06f30cd: Status 404 returned error can't find the container with id 357226bf45d1a5aaf86ef15a48abc03e77f9b596fa4f43f44043330fe06f30cd
May 27 23:43:16.866555 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:16.866526    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw"]
May 27 23:43:16.869631 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:16.869604    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5194cbd_67e6_43a3_8698_91ce60449345.slice/crio-c382d9d9494e7be64395b016d4c9dce78ceb4dc90b89209f6340a4c4ff9ed9bc WatchSource:0}: Error finding container c382d9d9494e7be64395b016d4c9dce78ceb4dc90b89209f6340a4c4ff9ed9bc: Status 404 returned error can't find the container with id c382d9d9494e7be64395b016d4c9dce78ceb4dc90b89209f6340a4c4ff9ed9bc
May 27 23:43:17.560512 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:17.560471    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw" event={"ID":"d5194cbd-67e6-43a3-8698-91ce60449345","Type":"ContainerStarted","Data":"c382d9d9494e7be64395b016d4c9dce78ceb4dc90b89209f6340a4c4ff9ed9bc"}
May 27 23:43:17.561728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:17.561696    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm" event={"ID":"4717a5f4-dc35-49e8-a0f1-b2e008417733","Type":"ContainerStarted","Data":"357226bf45d1a5aaf86ef15a48abc03e77f9b596fa4f43f44043330fe06f30cd"}
May 27 23:43:17.871400 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:17.871301    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:17.871797 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:17.871428    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:43:17.871797 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:17.871502    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls podName:fddfa466-8322-4bb5-a94f-ad100bb45c81 nodeName:}" failed. No retries permitted until 2026-05-27 23:43:21.871480056 +0000 UTC m=+151.260983854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls") pod "cluster-samples-operator-697587599-64b5x" (UID: "fddfa466-8322-4bb5-a94f-ad100bb45c81") : secret "samples-operator-tls" not found
May 27 23:43:19.567785 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:19.567707    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw" event={"ID":"d5194cbd-67e6-43a3-8698-91ce60449345","Type":"ContainerStarted","Data":"027933029f32ba6cd6c01adea82a79cf8d13f4879755a7796e56749713c25afa"}
May 27 23:43:19.569329 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:19.569286    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm" event={"ID":"4717a5f4-dc35-49e8-a0f1-b2e008417733","Type":"ContainerStarted","Data":"2665c4bd8a216aae089bce7eb963c06097dc090dade06ff45df6c402d2090610"}
May 27 23:43:19.584862 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:19.584811    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw" podStartSLOduration=1.196543085 podStartE2EDuration="3.584795198s" podCreationTimestamp="2026-05-27 23:43:16 +0000 UTC" firstStartedPulling="2026-05-27 23:43:16.871462244 +0000 UTC m=+146.260966037" lastFinishedPulling="2026-05-27 23:43:19.259714358 +0000 UTC m=+148.649218150" observedRunningTime="2026-05-27 23:43:19.583773302 +0000 UTC m=+148.973277120" watchObservedRunningTime="2026-05-27 23:43:19.584795198 +0000 UTC m=+148.974299013"
May 27 23:43:19.604672 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:19.604622    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm" podStartSLOduration=1.107122777 podStartE2EDuration="3.604610824s" podCreationTimestamp="2026-05-27 23:43:16 +0000 UTC" firstStartedPulling="2026-05-27 23:43:16.759514187 +0000 UTC m=+146.149017982" lastFinishedPulling="2026-05-27 23:43:19.257002234 +0000 UTC m=+148.646506029" observedRunningTime="2026-05-27 23:43:19.603819954 +0000 UTC m=+148.993323767" watchObservedRunningTime="2026-05-27 23:43:19.604610824 +0000 UTC m=+148.994114638"
May 27 23:43:21.902615 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:21.902583    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:21.902963 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:21.902696    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 27 23:43:21.902963 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:21.902758    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls podName:fddfa466-8322-4bb5-a94f-ad100bb45c81 nodeName:}" failed. No retries permitted until 2026-05-27 23:43:29.902743713 +0000 UTC m=+159.292247505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls") pod "cluster-samples-operator-697587599-64b5x" (UID: "fddfa466-8322-4bb5-a94f-ad100bb45c81") : secret "samples-operator-tls" not found
May 27 23:43:23.648791 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.648763    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h6jgs"]
May 27 23:43:23.651977 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.651960    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.654598 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.654574    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 27 23:43:23.654726 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.654627    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hw44\""
May 27 23:43:23.654726 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.654633    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 27 23:43:23.655458 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.655440    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 27 23:43:23.655517 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.655476    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 27 23:43:23.667160 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.667136    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h6jgs"]
May 27 23:43:23.817683 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.817656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7877bede-c140-4014-bbd1-c9d71fb5728f-crio-socket\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.817874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.817705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsddj\" (UniqueName: \"kubernetes.io/projected/7877bede-c140-4014-bbd1-c9d71fb5728f-kube-api-access-wsddj\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.817874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.817731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7877bede-c140-4014-bbd1-c9d71fb5728f-data-volume\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.817874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.817799    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7877bede-c140-4014-bbd1-c9d71fb5728f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.817874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.817840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918219 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918190    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7877bede-c140-4014-bbd1-c9d71fb5728f-crio-socket\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918389 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918243    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wsddj\" (UniqueName: \"kubernetes.io/projected/7877bede-c140-4014-bbd1-c9d71fb5728f-kube-api-access-wsddj\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918389 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918342    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7877bede-c140-4014-bbd1-c9d71fb5728f-crio-socket\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918389 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918364    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7877bede-c140-4014-bbd1-c9d71fb5728f-data-volume\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918492 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918416    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7877bede-c140-4014-bbd1-c9d71fb5728f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918492 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918440    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918565 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:23.918551    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 23:43:23.918617 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918602    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7877bede-c140-4014-bbd1-c9d71fb5728f-data-volume\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.918652 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:23.918610    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls podName:7877bede-c140-4014-bbd1-c9d71fb5728f nodeName:}" failed. No retries permitted until 2026-05-27 23:43:24.418597107 +0000 UTC m=+153.808100899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6jgs" (UID: "7877bede-c140-4014-bbd1-c9d71fb5728f") : secret "insights-runtime-extractor-tls" not found
May 27 23:43:23.918926 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.918909    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7877bede-c140-4014-bbd1-c9d71fb5728f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:23.927588 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:23.927570    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsddj\" (UniqueName: \"kubernetes.io/projected/7877bede-c140-4014-bbd1-c9d71fb5728f-kube-api-access-wsddj\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:24.422971 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:24.422948    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:24.423125 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:24.423051    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 23:43:24.423125 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:24.423103    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls podName:7877bede-c140-4014-bbd1-c9d71fb5728f nodeName:}" failed. No retries permitted until 2026-05-27 23:43:25.423088326 +0000 UTC m=+154.812592118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6jgs" (UID: "7877bede-c140-4014-bbd1-c9d71fb5728f") : secret "insights-runtime-extractor-tls" not found
May 27 23:43:25.430726 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:25.430688    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:25.431102 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:25.430841    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 23:43:25.431102 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:25.430934    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls podName:7877bede-c140-4014-bbd1-c9d71fb5728f nodeName:}" failed. No retries permitted until 2026-05-27 23:43:27.430917547 +0000 UTC m=+156.820421345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6jgs" (UID: "7877bede-c140-4014-bbd1-c9d71fb5728f") : secret "insights-runtime-extractor-tls" not found
May 27 23:43:27.446301 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:27.446267    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:27.446706 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:27.446427    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 23:43:27.446706 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:27.446505    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls podName:7877bede-c140-4014-bbd1-c9d71fb5728f nodeName:}" failed. No retries permitted until 2026-05-27 23:43:31.446489488 +0000 UTC m=+160.835993280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6jgs" (UID: "7877bede-c140-4014-bbd1-c9d71fb5728f") : secret "insights-runtime-extractor-tls" not found
May 27 23:43:27.991755 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:27.991719    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff"
May 27 23:43:28.006890 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:28.006862    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-44grr" podUID="b39519dd-a77f-4429-913c-18762f94546d"
May 27 23:43:28.023477 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:28.023450    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-22265" podUID="30d6a16f-881b-4b68-892c-e306fd38659b"
May 27 23:43:28.586464 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:28.586435    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:43:28.586464 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:28.586466    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44grr"
May 27 23:43:29.232112 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:29.232076    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bp6lz" podUID="80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f"
May 27 23:43:29.965143 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:29.965100    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:29.967634 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:29.967607    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddfa466-8322-4bb5-a94f-ad100bb45c81-samples-operator-tls\") pod \"cluster-samples-operator-697587599-64b5x\" (UID: \"fddfa466-8322-4bb5-a94f-ad100bb45c81\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:30.061356 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:30.061333    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"
May 27 23:43:30.183362 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:30.183328    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x"]
May 27 23:43:30.590977 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:30.590882    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x" event={"ID":"fddfa466-8322-4bb5-a94f-ad100bb45c81","Type":"ContainerStarted","Data":"5d216f2ecfd8cba6acc6f1639bfe66ff1ff9f1fa9238dc91024a81c52cf05ac6"}
May 27 23:43:31.477740 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:31.477655    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:31.478089 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:31.477803    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 23:43:31.478089 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:31.477876    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls podName:7877bede-c140-4014-bbd1-c9d71fb5728f nodeName:}" failed. No retries permitted until 2026-05-27 23:43:39.477857753 +0000 UTC m=+168.867361565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6jgs" (UID: "7877bede-c140-4014-bbd1-c9d71fb5728f") : secret "insights-runtime-extractor-tls" not found
May 27 23:43:32.599426 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.599387    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x" event={"ID":"fddfa466-8322-4bb5-a94f-ad100bb45c81","Type":"ContainerStarted","Data":"788ce1ed1d97f6489b0a146c078bd93b3d473ce124a1039b9ed3aa34c53dc3e6"}
May 27 23:43:32.599426 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.599425    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x" event={"ID":"fddfa466-8322-4bb5-a94f-ad100bb45c81","Type":"ContainerStarted","Data":"e83868665032a5b13a01f5465d789b2b799337dc2b0d4ce3bc2bf867fc5e6824"}
May 27 23:43:32.618394 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.618350    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-64b5x" podStartSLOduration=16.960376655 podStartE2EDuration="18.618336655s" podCreationTimestamp="2026-05-27 23:43:14 +0000 UTC" firstStartedPulling="2026-05-27 23:43:30.231061856 +0000 UTC m=+159.620565649" lastFinishedPulling="2026-05-27 23:43:31.889021854 +0000 UTC m=+161.278525649" observedRunningTime="2026-05-27 23:43:32.617508874 +0000 UTC m=+162.007012687" watchObservedRunningTime="2026-05-27 23:43:32.618336655 +0000 UTC m=+162.007840468"
May 27 23:43:32.888511 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.888429    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:43:32.888511 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.888488    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:43:32.890865 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.890845    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39519dd-a77f-4429-913c-18762f94546d-metrics-tls\") pod \"dns-default-44grr\" (UID: \"b39519dd-a77f-4429-913c-18762f94546d\") " pod="openshift-dns/dns-default-44grr"
May 27 23:43:32.891012 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.890991    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"image-registry-6d6847498c-gkxfs\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") " pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:43:32.989320 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.989266    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:43:32.991655 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:32.991626    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30d6a16f-881b-4b68-892c-e306fd38659b-cert\") pod \"ingress-canary-22265\" (UID: \"30d6a16f-881b-4b68-892c-e306fd38659b\") " pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:43:33.090142 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.090111    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mpjmq\""
May 27 23:43:33.090142 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.090111    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rf4mt\""
May 27 23:43:33.097158 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.097144    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:43:33.097242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.097227    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44grr"
May 27 23:43:33.229805 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.229705    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-44grr"]
May 27 23:43:33.233586 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:33.233559    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39519dd_a77f_4429_913c_18762f94546d.slice/crio-62daecbc353f98d73f8cc0bb99728d8d18ded930858d22dfde202dd2afda9bc4 WatchSource:0}: Error finding container 62daecbc353f98d73f8cc0bb99728d8d18ded930858d22dfde202dd2afda9bc4: Status 404 returned error can't find the container with id 62daecbc353f98d73f8cc0bb99728d8d18ded930858d22dfde202dd2afda9bc4
May 27 23:43:33.247927 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.247900    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6847498c-gkxfs"]
May 27 23:43:33.250408 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:33.250374    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0b8634_6037_4df8_9cb0_a35ff9b57dff.slice/crio-cff37cd44b48e18ba07c059c75e501341281ce4f7136888aed8eb3d48cbf3c6d WatchSource:0}: Error finding container cff37cd44b48e18ba07c059c75e501341281ce4f7136888aed8eb3d48cbf3c6d: Status 404 returned error can't find the container with id cff37cd44b48e18ba07c059c75e501341281ce4f7136888aed8eb3d48cbf3c6d
May 27 23:43:33.604404 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.604369    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" event={"ID":"fe0b8634-6037-4df8-9cb0-a35ff9b57dff","Type":"ContainerStarted","Data":"bd223bb2f8092b7c68bdb521dceec254af1fe765f397c3f00089141e1f0f1b16"}
May 27 23:43:33.604404 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.604406    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" event={"ID":"fe0b8634-6037-4df8-9cb0-a35ff9b57dff","Type":"ContainerStarted","Data":"cff37cd44b48e18ba07c059c75e501341281ce4f7136888aed8eb3d48cbf3c6d"}
May 27 23:43:33.604898 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.604446    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:43:33.605563 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.605542    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44grr" event={"ID":"b39519dd-a77f-4429-913c-18762f94546d","Type":"ContainerStarted","Data":"62daecbc353f98d73f8cc0bb99728d8d18ded930858d22dfde202dd2afda9bc4"}
May 27 23:43:33.623905 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:33.623861    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" podStartSLOduration=180.623849 podStartE2EDuration="3m0.623849s" podCreationTimestamp="2026-05-27 23:40:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:43:33.623068743 +0000 UTC m=+163.012572569" watchObservedRunningTime="2026-05-27 23:43:33.623849 +0000 UTC m=+163.013352813"
May 27 23:43:34.610350 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:34.610265    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44grr" event={"ID":"b39519dd-a77f-4429-913c-18762f94546d","Type":"ContainerStarted","Data":"7da5dc567a82b2c057c2468ad34de08cd515cca0444b316c19406affe004adca"}
May 27 23:43:34.610350 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:34.610341    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44grr" event={"ID":"b39519dd-a77f-4429-913c-18762f94546d","Type":"ContainerStarted","Data":"35ed24b6e15b478de9d35e86cd11df1e33ec0bb2633d460b2ca754bb1906120d"}
May 27 23:43:34.636932 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:34.636854    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-44grr" podStartSLOduration=129.43528305 podStartE2EDuration="2m10.636843228s" podCreationTimestamp="2026-05-27 23:41:24 +0000 UTC" firstStartedPulling="2026-05-27 23:43:33.235945058 +0000 UTC m=+162.625448856" lastFinishedPulling="2026-05-27 23:43:34.437505242 +0000 UTC m=+163.827009034" observedRunningTime="2026-05-27 23:43:34.635700306 +0000 UTC m=+164.025204122" watchObservedRunningTime="2026-05-27 23:43:34.636843228 +0000 UTC m=+164.026347042"
May 27 23:43:35.613191 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:35.613161    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-44grr"
May 27 23:43:39.215787 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.215753    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:43:39.218906 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.218882    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2gd8z\""
May 27 23:43:39.226051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.226035    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22265"
May 27 23:43:39.345901 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.345872    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-22265"]
May 27 23:43:39.349229 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:39.349202    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d6a16f_881b_4b68_892c_e306fd38659b.slice/crio-af128ebf390d6a03965ea97d68dcd100673feb8d336f061b155dfffe6a270a34 WatchSource:0}: Error finding container af128ebf390d6a03965ea97d68dcd100673feb8d336f061b155dfffe6a270a34: Status 404 returned error can't find the container with id af128ebf390d6a03965ea97d68dcd100673feb8d336f061b155dfffe6a270a34
May 27 23:43:39.539073 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.538985    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:39.541490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.541470    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7877bede-c140-4014-bbd1-c9d71fb5728f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jgs\" (UID: \"7877bede-c140-4014-bbd1-c9d71fb5728f\") " pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:39.560439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.560411    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h6jgs"
May 27 23:43:39.626687 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.626653    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-22265" event={"ID":"30d6a16f-881b-4b68-892c-e306fd38659b","Type":"ContainerStarted","Data":"af128ebf390d6a03965ea97d68dcd100673feb8d336f061b155dfffe6a270a34"}
May 27 23:43:39.683530 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:39.683495    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h6jgs"]
May 27 23:43:39.686418 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:39.686385    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7877bede_c140_4014_bbd1_c9d71fb5728f.slice/crio-9ab8fb7d3807bf8292187d240c2d943a75789cdbc89e17c6e6500915983567cf WatchSource:0}: Error finding container 9ab8fb7d3807bf8292187d240c2d943a75789cdbc89e17c6e6500915983567cf: Status 404 returned error can't find the container with id 9ab8fb7d3807bf8292187d240c2d943a75789cdbc89e17c6e6500915983567cf
May 27 23:43:40.630474 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:40.630434    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jgs" event={"ID":"7877bede-c140-4014-bbd1-c9d71fb5728f","Type":"ContainerStarted","Data":"948b8f00b76339285864da9882fa98336280a1d4d2a0d84f361a3e309b24d503"}
May 27 23:43:40.630474 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:40.630475    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jgs" event={"ID":"7877bede-c140-4014-bbd1-c9d71fb5728f","Type":"ContainerStarted","Data":"9ab8fb7d3807bf8292187d240c2d943a75789cdbc89e17c6e6500915983567cf"}
May 27 23:43:41.635978 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:41.635928    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-22265" event={"ID":"30d6a16f-881b-4b68-892c-e306fd38659b","Type":"ContainerStarted","Data":"60fde9acf9a0a9b326a3016db7973aefbf4c9b188a1dbc4d14932f24641cf06a"}
May 27 23:43:41.637970 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:41.637941    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jgs" event={"ID":"7877bede-c140-4014-bbd1-c9d71fb5728f","Type":"ContainerStarted","Data":"d623d73f87680ec5893951a277fea9bbe929b7406bc021f83b4ce3635ecf7be6"}
May 27 23:43:41.654850 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:41.654810    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-22265" podStartSLOduration=136.054048253 podStartE2EDuration="2m17.654794123s" podCreationTimestamp="2026-05-27 23:41:24 +0000 UTC" firstStartedPulling="2026-05-27 23:43:39.351508039 +0000 UTC m=+168.741011831" lastFinishedPulling="2026-05-27 23:43:40.952253906 +0000 UTC m=+170.341757701" observedRunningTime="2026-05-27 23:43:41.65422121 +0000 UTC m=+171.043725024" watchObservedRunningTime="2026-05-27 23:43:41.654794123 +0000 UTC m=+171.044298134"
May 27 23:43:42.641940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:42.641899    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jgs" event={"ID":"7877bede-c140-4014-bbd1-c9d71fb5728f","Type":"ContainerStarted","Data":"e71fcf45ca9d5670fe5928c6fe9c532ddfc17fdba68830083633ffa0492a55e3"}
May 27 23:43:42.666226 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:42.666183    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h6jgs" podStartSLOduration=17.260252672 podStartE2EDuration="19.666170673s" podCreationTimestamp="2026-05-27 23:43:23 +0000 UTC" firstStartedPulling="2026-05-27 23:43:39.740746383 +0000 UTC m=+169.130250176" lastFinishedPulling="2026-05-27 23:43:42.146664368 +0000 UTC m=+171.536168177" observedRunningTime="2026-05-27 23:43:42.664980885 +0000 UTC m=+172.054484698" watchObservedRunningTime="2026-05-27 23:43:42.666170673 +0000 UTC m=+172.055674486"
May 27 23:43:43.562008 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.561966    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-v28ps"]
May 27 23:43:43.565126 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.565104    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:43:43.567578 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.567560    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g5rgh\""
May 27 23:43:43.567838 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.567821    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 27 23:43:43.567921 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.567906    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 27 23:43:43.574990 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.574967    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-v28ps"]
May 27 23:43:43.577967 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.577943    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6847498c-gkxfs"]
May 27 23:43:43.669470 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.669440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgcr\" (UniqueName: \"kubernetes.io/projected/d22b1e67-08e2-4858-871d-7a64ef7b09d5-kube-api-access-gbgcr\") pod \"downloads-57bd96b888-v28ps\" (UID: \"d22b1e67-08e2-4858-871d-7a64ef7b09d5\") " pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:43:43.669880 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.669865    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d555f9d9c-vcntc"]
May 27 23:43:43.672834 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.672818    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"]
May 27 23:43:43.672969 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.672952    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.675757 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.675734    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:43.678284 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.678265    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vr7l7\""
May 27 23:43:43.678558 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.678540    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 27 23:43:43.683596 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.683579    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"]
May 27 23:43:43.686834 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.686816    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d555f9d9c-vcntc"]
May 27 23:43:43.770508 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43474b03-1e8a-4922-9b6d-c6b4c1291521-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-cjx69\" (UID: \"43474b03-1e8a-4922-9b6d-c6b4c1291521\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:43.770652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77242b73-dafd-43f9-b91c-ca07a5a7f56a-installation-pull-secrets\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770537    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-bound-sa-token\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77242b73-dafd-43f9-b91c-ca07a5a7f56a-trusted-ca\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cvbk\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-kube-api-access-8cvbk\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770622    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77242b73-dafd-43f9-b91c-ca07a5a7f56a-registry-certificates\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770814 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770695    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77242b73-dafd-43f9-b91c-ca07a5a7f56a-ca-trust-extracted\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770814 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-registry-tls\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770814 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77242b73-dafd-43f9-b91c-ca07a5a7f56a-image-registry-private-configuration\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.770947 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.770897    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgcr\" (UniqueName: \"kubernetes.io/projected/d22b1e67-08e2-4858-871d-7a64ef7b09d5-kube-api-access-gbgcr\") pod \"downloads-57bd96b888-v28ps\" (UID: \"d22b1e67-08e2-4858-871d-7a64ef7b09d5\") " pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:43:43.781356 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.781334    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgcr\" (UniqueName: \"kubernetes.io/projected/d22b1e67-08e2-4858-871d-7a64ef7b09d5-kube-api-access-gbgcr\") pod \"downloads-57bd96b888-v28ps\" (UID: \"d22b1e67-08e2-4858-871d-7a64ef7b09d5\") " pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:43:43.871898 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.871830    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8cvbk\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-kube-api-access-8cvbk\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.871898 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.871858    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77242b73-dafd-43f9-b91c-ca07a5a7f56a-registry-certificates\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.871898 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.871885    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77242b73-dafd-43f9-b91c-ca07a5a7f56a-ca-trust-extracted\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.871910    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-registry-tls\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.871938    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77242b73-dafd-43f9-b91c-ca07a5a7f56a-image-registry-private-configuration\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.871996    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43474b03-1e8a-4922-9b6d-c6b4c1291521-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-cjx69\" (UID: \"43474b03-1e8a-4922-9b6d-c6b4c1291521\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:43.872110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.872022    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77242b73-dafd-43f9-b91c-ca07a5a7f56a-installation-pull-secrets\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.872046    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-bound-sa-token\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.872072    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77242b73-dafd-43f9-b91c-ca07a5a7f56a-trusted-ca\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872783 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.872762    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77242b73-dafd-43f9-b91c-ca07a5a7f56a-ca-trust-extracted\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.872989 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.872968    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77242b73-dafd-43f9-b91c-ca07a5a7f56a-registry-certificates\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.873081 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.872983    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77242b73-dafd-43f9-b91c-ca07a5a7f56a-trusted-ca\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.873582 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.873561    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:43:43.874707 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.874687    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-registry-tls\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.874796 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.874776    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43474b03-1e8a-4922-9b6d-c6b4c1291521-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-cjx69\" (UID: \"43474b03-1e8a-4922-9b6d-c6b4c1291521\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:43.874996 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.874979    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77242b73-dafd-43f9-b91c-ca07a5a7f56a-image-registry-private-configuration\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.875148 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.875125    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77242b73-dafd-43f9-b91c-ca07a5a7f56a-installation-pull-secrets\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.879633 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.879612    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-bound-sa-token\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.879771 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.879755    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cvbk\" (UniqueName: \"kubernetes.io/projected/77242b73-dafd-43f9-b91c-ca07a5a7f56a-kube-api-access-8cvbk\") pod \"image-registry-6d555f9d9c-vcntc\" (UID: \"77242b73-dafd-43f9-b91c-ca07a5a7f56a\") " pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.982869 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.982836    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:43.989360 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:43.988610    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:44.029127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.029076    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-v28ps"]
May 27 23:43:44.034510 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:44.034482    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22b1e67_08e2_4858_871d_7a64ef7b09d5.slice/crio-1a8edff9e6ac4a31bc222a6c6d437c00e6c71da256b4c71f9e306824c02e1360 WatchSource:0}: Error finding container 1a8edff9e6ac4a31bc222a6c6d437c00e6c71da256b4c71f9e306824c02e1360: Status 404 returned error can't find the container with id 1a8edff9e6ac4a31bc222a6c6d437c00e6c71da256b4c71f9e306824c02e1360
May 27 23:43:44.122870 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.122777    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d555f9d9c-vcntc"]
May 27 23:43:44.126438 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:44.126410    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77242b73_dafd_43f9_b91c_ca07a5a7f56a.slice/crio-1447da946ac1cc0fbc10614dc0dbd285c34c9e56c6fb58d6416eaea0c6e63918 WatchSource:0}: Error finding container 1447da946ac1cc0fbc10614dc0dbd285c34c9e56c6fb58d6416eaea0c6e63918: Status 404 returned error can't find the container with id 1447da946ac1cc0fbc10614dc0dbd285c34c9e56c6fb58d6416eaea0c6e63918
May 27 23:43:44.134843 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.134822    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"]
May 27 23:43:44.138689 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:44.138664    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43474b03_1e8a_4922_9b6d_c6b4c1291521.slice/crio-7446067e6ab7eda55359ddb808e083eacf144eb3ffaf334fc7019ae24ec06cd2 WatchSource:0}: Error finding container 7446067e6ab7eda55359ddb808e083eacf144eb3ffaf334fc7019ae24ec06cd2: Status 404 returned error can't find the container with id 7446067e6ab7eda55359ddb808e083eacf144eb3ffaf334fc7019ae24ec06cd2
May 27 23:43:44.215157 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.215132    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:43:44.648291 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.648252    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69" event={"ID":"43474b03-1e8a-4922-9b6d-c6b4c1291521","Type":"ContainerStarted","Data":"7446067e6ab7eda55359ddb808e083eacf144eb3ffaf334fc7019ae24ec06cd2"}
May 27 23:43:44.650098 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.650036    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc" event={"ID":"77242b73-dafd-43f9-b91c-ca07a5a7f56a","Type":"ContainerStarted","Data":"a9b8ccd97e9d0c17a18e17218924921acc8579da618fbe41366b43d9d773b78e"}
May 27 23:43:44.650098 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.650076    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc" event={"ID":"77242b73-dafd-43f9-b91c-ca07a5a7f56a","Type":"ContainerStarted","Data":"1447da946ac1cc0fbc10614dc0dbd285c34c9e56c6fb58d6416eaea0c6e63918"}
May 27 23:43:44.650281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.650165    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:43:44.651650 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.651621    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-v28ps" event={"ID":"d22b1e67-08e2-4858-871d-7a64ef7b09d5","Type":"ContainerStarted","Data":"1a8edff9e6ac4a31bc222a6c6d437c00e6c71da256b4c71f9e306824c02e1360"}
May 27 23:43:44.670749 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:44.670695    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc" podStartSLOduration=1.670675685 podStartE2EDuration="1.670675685s" podCreationTimestamp="2026-05-27 23:43:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:43:44.669332445 +0000 UTC m=+174.058836256" watchObservedRunningTime="2026-05-27 23:43:44.670675685 +0000 UTC m=+174.060179501"
May 27 23:43:45.618810 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:45.618738    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-44grr"
May 27 23:43:45.656042 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:45.656002    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69" event={"ID":"43474b03-1e8a-4922-9b6d-c6b4c1291521","Type":"ContainerStarted","Data":"ac25143494b91957387b8b8e3aefe8f714400fd33d181062fe15db7a39080de2"}
May 27 23:43:45.656271 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:45.656247    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:45.661793 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:45.661768    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69"
May 27 23:43:45.673411 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:45.673363    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-cjx69" podStartSLOduration=1.632245902 podStartE2EDuration="2.673351593s" podCreationTimestamp="2026-05-27 23:43:43 +0000 UTC" firstStartedPulling="2026-05-27 23:43:44.140366169 +0000 UTC m=+173.529869961" lastFinishedPulling="2026-05-27 23:43:45.181471846 +0000 UTC m=+174.570975652" observedRunningTime="2026-05-27 23:43:45.672486178 +0000 UTC m=+175.061989987" watchObservedRunningTime="2026-05-27 23:43:45.673351593 +0000 UTC m=+175.062855409"
May 27 23:43:46.779791 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.779757    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"]
May 27 23:43:46.783351 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.783328    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.787039 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.787013    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 27 23:43:46.787039 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.787034    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vs6wg\""
May 27 23:43:46.787233 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.787093    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 27 23:43:46.787233 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.787035    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 27 23:43:46.787421 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.787366    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 27 23:43:46.787482 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.787459    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 27 23:43:46.791251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.791231    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"]
May 27 23:43:46.893279 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.893246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.893448 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.893293    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.893448 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.893331    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.893448 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.893368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6cg\" (UniqueName: \"kubernetes.io/projected/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-kube-api-access-kt6cg\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.994363 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.994326    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.994504 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.994373    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.994504 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.994421    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6cg\" (UniqueName: \"kubernetes.io/projected/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-kube-api-access-kt6cg\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.994504 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.994482    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.995133 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.995108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.997524 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.997487    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:46.997874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:46.997854    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:47.004607 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:47.004483    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6cg\" (UniqueName: \"kubernetes.io/projected/37a0dd1e-ed46-462c-8b1d-c8a55c448f4d-kube-api-access-kt6cg\") pod \"prometheus-operator-7779bbdd78-9gddx\" (UID: \"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:47.095488 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:47.095407    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"
May 27 23:43:47.237747 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:47.237713    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-9gddx"]
May 27 23:43:47.240879 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:47.240849    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a0dd1e_ed46_462c_8b1d_c8a55c448f4d.slice/crio-78a5fa7b0f748f579c25702c60c6867bf18e834f694ef6ed03c28225a140a011 WatchSource:0}: Error finding container 78a5fa7b0f748f579c25702c60c6867bf18e834f694ef6ed03c28225a140a011: Status 404 returned error can't find the container with id 78a5fa7b0f748f579c25702c60c6867bf18e834f694ef6ed03c28225a140a011
May 27 23:43:47.663969 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:47.663879    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx" event={"ID":"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d","Type":"ContainerStarted","Data":"78a5fa7b0f748f579c25702c60c6867bf18e834f694ef6ed03c28225a140a011"}
May 27 23:43:48.669392 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:48.669336    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx" event={"ID":"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d","Type":"ContainerStarted","Data":"2a8a071c5ccbeba05e527629ec39beb99664a70f10ea5fdb581b97a9d16a5060"}
May 27 23:43:48.669392 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:48.669372    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx" event={"ID":"37a0dd1e-ed46-462c-8b1d-c8a55c448f4d","Type":"ContainerStarted","Data":"6626e7d454fee8c7b084057e5a7ac8b5391a705a4f928cdc5e8efa97be72ca4e"}
May 27 23:43:48.687983 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:48.687180    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9gddx" podStartSLOduration=1.422934103 podStartE2EDuration="2.687167317s" podCreationTimestamp="2026-05-27 23:43:46 +0000 UTC" firstStartedPulling="2026-05-27 23:43:47.242870248 +0000 UTC m=+176.632374041" lastFinishedPulling="2026-05-27 23:43:48.507103458 +0000 UTC m=+177.896607255" observedRunningTime="2026-05-27 23:43:48.686448902 +0000 UTC m=+178.075952747" watchObservedRunningTime="2026-05-27 23:43:48.687167317 +0000 UTC m=+178.076671131"
May 27 23:43:50.187204 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.187145    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wdnfv"]
May 27 23:43:50.192354 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.192326    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.195522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.195498    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 27 23:43:50.196493 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.196453    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57dgk\""
May 27 23:43:50.197501 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.197484    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 27 23:43:50.197760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.197745    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 27 23:43:50.323192 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323161    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323390 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-textfile\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323390 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323238    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-tls\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323390 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-root\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323390 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd52a86d-0131-4b50-9a21-480c11028692-metrics-client-ca\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323390 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-sys\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323418    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-wtmp\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.323613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.323500    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hzv\" (UniqueName: \"kubernetes.io/projected/fd52a86d-0131-4b50-9a21-480c11028692-kube-api-access-c6hzv\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424145 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424116    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-sys\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424155    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-wtmp\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424185    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424232    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hzv\" (UniqueName: \"kubernetes.io/projected/fd52a86d-0131-4b50-9a21-480c11028692-kube-api-access-c6hzv\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424232    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-sys\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424265    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424296    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-textfile\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424333    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-wtmp\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424340    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-tls\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424399    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-root\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424438    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd52a86d-0131-4b50-9a21-480c11028692-metrics-client-ca\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424612    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd52a86d-0131-4b50-9a21-480c11028692-root\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424808 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424789    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-textfile\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.424961 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424940    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd52a86d-0131-4b50-9a21-480c11028692-metrics-client-ca\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.425028 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.424970    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.427156 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.427133    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-tls\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.427259 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.427160    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd52a86d-0131-4b50-9a21-480c11028692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.433939 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.433890    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hzv\" (UniqueName: \"kubernetes.io/projected/fd52a86d-0131-4b50-9a21-480c11028692-kube-api-access-c6hzv\") pod \"node-exporter-wdnfv\" (UID: \"fd52a86d-0131-4b50-9a21-480c11028692\") " pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.503932 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.503861    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wdnfv"
May 27 23:43:50.513123 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:50.513085    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd52a86d_0131_4b50_9a21_480c11028692.slice/crio-287b33251712431f4287f4f27774161a3f2ef2a53f572f0199f58e22c523bf3e WatchSource:0}: Error finding container 287b33251712431f4287f4f27774161a3f2ef2a53f572f0199f58e22c523bf3e: Status 404 returned error can't find the container with id 287b33251712431f4287f4f27774161a3f2ef2a53f572f0199f58e22c523bf3e
May 27 23:43:50.650060 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.649980    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6447759fb4-ftgrv"]
May 27 23:43:50.654527 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.654503    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.657198 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.657166    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 27 23:43:50.657345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.657245    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cdblb\""
May 27 23:43:50.657345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.657263    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 27 23:43:50.657345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.657287    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 27 23:43:50.657522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.657409    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 27 23:43:50.658270 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.658250    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 27 23:43:50.664625 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.664603    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6447759fb4-ftgrv"]
May 27 23:43:50.676381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.676300    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdnfv" event={"ID":"fd52a86d-0131-4b50-9a21-480c11028692","Type":"ContainerStarted","Data":"287b33251712431f4287f4f27774161a3f2ef2a53f572f0199f58e22c523bf3e"}
May 27 23:43:50.727460 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.727380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-oauth-serving-cert\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.727460 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.727435    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-oauth-config\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.727652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.727478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-config\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.727652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.727509    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjftd\" (UniqueName: \"kubernetes.io/projected/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-kube-api-access-pjftd\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.727652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.727570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-serving-cert\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.727652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.727615    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-service-ca\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.828652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.828567    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-oauth-config\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.828652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.828610    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-config\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.828652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.828637    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pjftd\" (UniqueName: \"kubernetes.io/projected/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-kube-api-access-pjftd\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.828920 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.828676    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-serving-cert\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.828920 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.828724    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-service-ca\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.828920 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.828777    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-oauth-serving-cert\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.829471 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.829443    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-config\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.829592 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.829489    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-oauth-serving-cert\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.829592 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.829589    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-service-ca\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.831599 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.831574    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-serving-cert\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.831783 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.831765    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-oauth-config\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.839991 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.839968    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjftd\" (UniqueName: \"kubernetes.io/projected/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-kube-api-access-pjftd\") pod \"console-6447759fb4-ftgrv\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") " pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:50.966653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:50.966617    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:43:51.125981 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:51.125905    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6447759fb4-ftgrv"]
May 27 23:43:51.130955 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:43:51.130921    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fce832b_e0dc_4c8e_9cc9_906010a0d5ea.slice/crio-b1adfdeb7c2bc4547bc81c1e1e8f0390ff972b4a9a1bf6a692ec99857e73ffbc WatchSource:0}: Error finding container b1adfdeb7c2bc4547bc81c1e1e8f0390ff972b4a9a1bf6a692ec99857e73ffbc: Status 404 returned error can't find the container with id b1adfdeb7c2bc4547bc81c1e1e8f0390ff972b4a9a1bf6a692ec99857e73ffbc
May 27 23:43:51.683726 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:51.683681    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdnfv" event={"ID":"fd52a86d-0131-4b50-9a21-480c11028692","Type":"ContainerStarted","Data":"45d1be6dfd4dba8780081b455402d28d58cbedeceb90afa91a4d719c0be09441"}
May 27 23:43:51.685833 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:51.685802    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447759fb4-ftgrv" event={"ID":"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea","Type":"ContainerStarted","Data":"b1adfdeb7c2bc4547bc81c1e1e8f0390ff972b4a9a1bf6a692ec99857e73ffbc"}
May 27 23:43:52.690198 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:52.690158    2572 generic.go:358] "Generic (PLEG): container finished" podID="fd52a86d-0131-4b50-9a21-480c11028692" containerID="45d1be6dfd4dba8780081b455402d28d58cbedeceb90afa91a4d719c0be09441" exitCode=0
May 27 23:43:52.690663 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:52.690223    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdnfv" event={"ID":"fd52a86d-0131-4b50-9a21-480c11028692","Type":"ContainerDied","Data":"45d1be6dfd4dba8780081b455402d28d58cbedeceb90afa91a4d719c0be09441"}
May 27 23:43:53.584435 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:53.584394    2572 patch_prober.go:28] interesting pod/image-registry-6d6847498c-gkxfs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 27 23:43:53.584606 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:53.584474    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 27 23:43:53.696515 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:53.696474    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdnfv" event={"ID":"fd52a86d-0131-4b50-9a21-480c11028692","Type":"ContainerStarted","Data":"6cce0e1a8b62201d3312b69260a2dd612ce562bb147e89a5c06304bdd2cf0435"}
May 27 23:43:53.696515 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:53.696517    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdnfv" event={"ID":"fd52a86d-0131-4b50-9a21-480c11028692","Type":"ContainerStarted","Data":"3f57bbc8b537a46335a66dca407bbcbb179956f8744e17359d04fb61d77073cc"}
May 27 23:43:53.719928 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:53.719870    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wdnfv" podStartSLOduration=2.719781056 podStartE2EDuration="3.719850786s" podCreationTimestamp="2026-05-27 23:43:50 +0000 UTC" firstStartedPulling="2026-05-27 23:43:50.514918871 +0000 UTC m=+179.904422665" lastFinishedPulling="2026-05-27 23:43:51.514988595 +0000 UTC m=+180.904492395" observedRunningTime="2026-05-27 23:43:53.71933867 +0000 UTC m=+183.108842485" watchObservedRunningTime="2026-05-27 23:43:53.719850786 +0000 UTC m=+183.109354600"
May 27 23:43:54.815229 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.815187    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f4857778d-szdtm"]
May 27 23:43:54.817973 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.817948    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.821586 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.821559    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kh49k\""
May 27 23:43:54.821858 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.821843    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 27 23:43:54.822582 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.822560    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 27 23:43:54.822810 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.822790    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 27 23:43:54.822911 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.822886    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3o9uo3itcn6l8\""
May 27 23:43:54.822970 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.822790    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 27 23:43:54.831150 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.831127    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f4857778d-szdtm"]
May 27 23:43:54.866914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.866882    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-secret-metrics-server-client-certs\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.867086 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.866936    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-client-ca-bundle\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.867086 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.867010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tfc\" (UniqueName: \"kubernetes.io/projected/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-kube-api-access-f9tfc\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.867204 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.867103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.867204 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.867170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-audit-log\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.867204 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.867196    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-secret-metrics-server-tls\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.867381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.867216    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-metrics-server-audit-profiles\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.937005 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.936975    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"]
May 27 23:43:54.939896 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.939780    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:43:54.942840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.942670    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 27 23:43:54.942840 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.942783    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-4dq9d\""
May 27 23:43:54.948024 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.948001    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"]
May 27 23:43:54.968629 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968600    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.968781 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968690    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-audit-log\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.968781 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968726    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-secret-metrics-server-tls\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.968781 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968752    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-metrics-server-audit-profiles\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.968781 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968778    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-secret-metrics-server-client-certs\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.968951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968800    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-client-ca-bundle\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.968951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.968829    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tfc\" (UniqueName: \"kubernetes.io/projected/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-kube-api-access-f9tfc\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.969448 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.969424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-audit-log\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.969525 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.969460    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.970894 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.970849    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-metrics-server-audit-profiles\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.972236 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.972211    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-secret-metrics-server-tls\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.972236 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.972227    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-secret-metrics-server-client-certs\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.972798 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.972775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-client-ca-bundle\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:54.979480 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:54.979440    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tfc\" (UniqueName: \"kubernetes.io/projected/b1bbc942-d7f9-42c0-9f46-28de571bbcf7-kube-api-access-f9tfc\") pod \"metrics-server-6f4857778d-szdtm\" (UID: \"b1bbc942-d7f9-42c0-9f46-28de571bbcf7\") " pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:55.069806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:55.069720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01319e4a-e3e5-4e1e-b0e5-b0efa5113687-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sfj7f\" (UID: \"01319e4a-e3e5-4e1e-b0e5-b0efa5113687\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:43:55.130194 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:55.130166    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:43:55.170827 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:55.170794    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01319e4a-e3e5-4e1e-b0e5-b0efa5113687-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sfj7f\" (UID: \"01319e4a-e3e5-4e1e-b0e5-b0efa5113687\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:43:55.171008 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:55.170946    2572 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 27 23:43:55.171082 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:43:55.171032    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01319e4a-e3e5-4e1e-b0e5-b0efa5113687-monitoring-plugin-cert podName:01319e4a-e3e5-4e1e-b0e5-b0efa5113687 nodeName:}" failed. No retries permitted until 2026-05-27 23:43:55.67101061 +0000 UTC m=+185.060514406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/01319e4a-e3e5-4e1e-b0e5-b0efa5113687-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-sfj7f" (UID: "01319e4a-e3e5-4e1e-b0e5-b0efa5113687") : secret "monitoring-plugin-cert" not found
May 27 23:43:55.674324 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:55.674266    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01319e4a-e3e5-4e1e-b0e5-b0efa5113687-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sfj7f\" (UID: \"01319e4a-e3e5-4e1e-b0e5-b0efa5113687\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:43:55.676892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:55.676864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01319e4a-e3e5-4e1e-b0e5-b0efa5113687-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-sfj7f\" (UID: \"01319e4a-e3e5-4e1e-b0e5-b0efa5113687\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:43:55.853159 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:55.853112    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:43:59.993151 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:59.993114    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d979df644-t4lqr"]
May 27 23:43:59.997331 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:43:59.997286    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.005416 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.005390    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 27 23:44:00.007988 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.007968    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d979df644-t4lqr"]
May 27 23:44:00.116688 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-console-config\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.116688 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116690    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-oauth-serving-cert\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.116922 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-service-ca\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.116922 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116794    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchzv\" (UniqueName: \"kubernetes.io/projected/8f213f5d-175c-46aa-8d10-920e8599cd07-kube-api-access-kchzv\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.116922 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116838    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-oauth-config\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.116922 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-trusted-ca-bundle\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.116922 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.116913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-serving-cert\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218165 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218124    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-serving-cert\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218338 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218195    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-console-config\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218338 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218242    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-oauth-serving-cert\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218338 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218328    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-service-ca\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218509 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218361    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kchzv\" (UniqueName: \"kubernetes.io/projected/8f213f5d-175c-46aa-8d10-920e8599cd07-kube-api-access-kchzv\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218509 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218394    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-oauth-config\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.218509 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.218421    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-trusted-ca-bundle\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.219096 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.219070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-oauth-serving-cert\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.219220 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.219070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-service-ca\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.219220 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.219127    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-console-config\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.219786 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.219762    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-trusted-ca-bundle\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.221208 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.221187    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-serving-cert\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.221301 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.221223    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-oauth-config\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.227994 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.227973    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchzv\" (UniqueName: \"kubernetes.io/projected/8f213f5d-175c-46aa-8d10-920e8599cd07-kube-api-access-kchzv\") pod \"console-7d979df644-t4lqr\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") " pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:00.309486 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:00.309393    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:01.724551 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:01.724357    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447759fb4-ftgrv" event={"ID":"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea","Type":"ContainerStarted","Data":"8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476"}
May 27 23:44:01.766088 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:01.765937    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6447759fb4-ftgrv" podStartSLOduration=1.3035626790000001 podStartE2EDuration="11.765918188s" podCreationTimestamp="2026-05-27 23:43:50 +0000 UTC" firstStartedPulling="2026-05-27 23:43:51.133918464 +0000 UTC m=+180.523422260" lastFinishedPulling="2026-05-27 23:44:01.596273964 +0000 UTC m=+190.985777769" observedRunningTime="2026-05-27 23:44:01.758834969 +0000 UTC m=+191.148338784" watchObservedRunningTime="2026-05-27 23:44:01.765918188 +0000 UTC m=+191.155422002"
May 27 23:44:01.767110 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:44:01.767068    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f213f5d_175c_46aa_8d10_920e8599cd07.slice/crio-dc1e815f4a8b8f530f2727340bf5ecffc3fdfb6ac8be709609ecd9b4a6a5a4ef WatchSource:0}: Error finding container dc1e815f4a8b8f530f2727340bf5ecffc3fdfb6ac8be709609ecd9b4a6a5a4ef: Status 404 returned error can't find the container with id dc1e815f4a8b8f530f2727340bf5ecffc3fdfb6ac8be709609ecd9b4a6a5a4ef
May 27 23:44:01.768222 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:01.768124    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d979df644-t4lqr"]
May 27 23:44:01.784393 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:01.784359    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f4857778d-szdtm"]
May 27 23:44:01.788396 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:44:01.788356    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bbc942_d7f9_42c0_9f46_28de571bbcf7.slice/crio-c061bc2f7eff9b3c3b34a1569f3a9781309b9ff7b8d03ec18e900e160165f404 WatchSource:0}: Error finding container c061bc2f7eff9b3c3b34a1569f3a9781309b9ff7b8d03ec18e900e160165f404: Status 404 returned error can't find the container with id c061bc2f7eff9b3c3b34a1569f3a9781309b9ff7b8d03ec18e900e160165f404
May 27 23:44:01.800995 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:01.800968    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"]
May 27 23:44:01.807511 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:44:01.807475    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01319e4a_e3e5_4e1e_b0e5_b0efa5113687.slice/crio-11099ccad6d4092d8690b25cc438ce6d4515bb033785bf401a438fa6d72ece9e WatchSource:0}: Error finding container 11099ccad6d4092d8690b25cc438ce6d4515bb033785bf401a438fa6d72ece9e: Status 404 returned error can't find the container with id 11099ccad6d4092d8690b25cc438ce6d4515bb033785bf401a438fa6d72ece9e
May 27 23:44:02.733918 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.733882    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-v28ps" event={"ID":"d22b1e67-08e2-4858-871d-7a64ef7b09d5","Type":"ContainerStarted","Data":"6b06f869cc780f6bbb5fa74f11d2cd1b9463011cbaf2bd26a1794f4ceb390740"}
May 27 23:44:02.735347 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.735299    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:44:02.737410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.737381    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f" event={"ID":"01319e4a-e3e5-4e1e-b0e5-b0efa5113687","Type":"ContainerStarted","Data":"11099ccad6d4092d8690b25cc438ce6d4515bb033785bf401a438fa6d72ece9e"}
May 27 23:44:02.739043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.739016    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d979df644-t4lqr" event={"ID":"8f213f5d-175c-46aa-8d10-920e8599cd07","Type":"ContainerStarted","Data":"f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d"}
May 27 23:44:02.739160 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.739047    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d979df644-t4lqr" event={"ID":"8f213f5d-175c-46aa-8d10-920e8599cd07","Type":"ContainerStarted","Data":"dc1e815f4a8b8f530f2727340bf5ecffc3fdfb6ac8be709609ecd9b4a6a5a4ef"}
May 27 23:44:02.740894 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.740857    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm" event={"ID":"b1bbc942-d7f9-42c0-9f46-28de571bbcf7","Type":"ContainerStarted","Data":"c061bc2f7eff9b3c3b34a1569f3a9781309b9ff7b8d03ec18e900e160165f404"}
May 27 23:44:02.749424 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.749400    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-v28ps"
May 27 23:44:02.758835 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:02.758782    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-v28ps" podStartSLOduration=2.134603572 podStartE2EDuration="19.758764959s" podCreationTimestamp="2026-05-27 23:43:43 +0000 UTC" firstStartedPulling="2026-05-27 23:43:44.037568441 +0000 UTC m=+173.427072232" lastFinishedPulling="2026-05-27 23:44:01.661729824 +0000 UTC m=+191.051233619" observedRunningTime="2026-05-27 23:44:02.756013678 +0000 UTC m=+192.145517533" watchObservedRunningTime="2026-05-27 23:44:02.758764959 +0000 UTC m=+192.148268774"
May 27 23:44:03.584092 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:03.583857    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:44:03.609680 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:03.609619    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7d979df644-t4lqr" podStartSLOduration=4.609598338 podStartE2EDuration="4.609598338s" podCreationTimestamp="2026-05-27 23:43:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:44:02.79810963 +0000 UTC m=+192.187613443" watchObservedRunningTime="2026-05-27 23:44:03.609598338 +0000 UTC m=+192.999102153"
May 27 23:44:04.752527 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:04.752109    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f" event={"ID":"01319e4a-e3e5-4e1e-b0e5-b0efa5113687","Type":"ContainerStarted","Data":"a5266e90743e6ab4774e81df9dc04c0976a1baf8caa2d2e7101b351fd8b402a7"}
May 27 23:44:04.752527 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:04.752516    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:44:04.754556 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:04.754472    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm" event={"ID":"b1bbc942-d7f9-42c0-9f46-28de571bbcf7","Type":"ContainerStarted","Data":"26b67b7b666082de256836e06d35605e002807029445db993e061c86879b3d9c"}
May 27 23:44:04.759825 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:04.759790    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f"
May 27 23:44:04.769601 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:04.769555    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-sfj7f" podStartSLOduration=8.002597777 podStartE2EDuration="10.769541837s" podCreationTimestamp="2026-05-27 23:43:54 +0000 UTC" firstStartedPulling="2026-05-27 23:44:01.809922549 +0000 UTC m=+191.199426341" lastFinishedPulling="2026-05-27 23:44:04.576866599 +0000 UTC m=+193.966370401" observedRunningTime="2026-05-27 23:44:04.768804702 +0000 UTC m=+194.158308519" watchObservedRunningTime="2026-05-27 23:44:04.769541837 +0000 UTC m=+194.159045652"
May 27 23:44:04.804045 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:04.803936    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm" podStartSLOduration=8.014704076 podStartE2EDuration="10.80391785s" podCreationTimestamp="2026-05-27 23:43:54 +0000 UTC" firstStartedPulling="2026-05-27 23:44:01.790858365 +0000 UTC m=+191.180362160" lastFinishedPulling="2026-05-27 23:44:04.580072143 +0000 UTC m=+193.969575934" observedRunningTime="2026-05-27 23:44:04.802573193 +0000 UTC m=+194.192077144" watchObservedRunningTime="2026-05-27 23:44:04.80391785 +0000 UTC m=+194.193421665"
May 27 23:44:05.661127 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:05.661095    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d555f9d9c-vcntc"
May 27 23:44:08.597156 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:08.597070    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" containerName="registry" containerID="cri-o://bd223bb2f8092b7c68bdb521dceec254af1fe765f397c3f00089141e1f0f1b16" gracePeriod=30
May 27 23:44:08.769371 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:08.769234    2572 generic.go:358] "Generic (PLEG): container finished" podID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" containerID="bd223bb2f8092b7c68bdb521dceec254af1fe765f397c3f00089141e1f0f1b16" exitCode=0
May 27 23:44:08.769371 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:08.769356    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" event={"ID":"fe0b8634-6037-4df8-9cb0-a35ff9b57dff","Type":"ContainerDied","Data":"bd223bb2f8092b7c68bdb521dceec254af1fe765f397c3f00089141e1f0f1b16"}
May 27 23:44:08.932091 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:08.931947    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:44:09.109166 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109122    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-trusted-ca\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109373 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109213    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109373 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109287    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-certificates\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109373 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109342    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-bound-sa-token\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109373 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109369    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-ca-trust-extracted\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109605 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109395    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75f8h\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-kube-api-access-75f8h\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109605 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109443    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-image-registry-private-configuration\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109605 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109468    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-installation-pull-secrets\") pod \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\" (UID: \"fe0b8634-6037-4df8-9cb0-a35ff9b57dff\") "
May 27 23:44:09.109761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109647    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:44:09.109761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109715    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:44:09.109863 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109812    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-certificates\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.109863 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.109833    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-trusted-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.117788 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.117729    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:09.118151 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.118113    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-kube-api-access-75f8h" (OuterVolumeSpecName: "kube-api-access-75f8h") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "kube-api-access-75f8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:44:09.118272 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.118226    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:44:09.118594 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.118563    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:44:09.118717 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.118680    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:09.122292 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.122258    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fe0b8634-6037-4df8-9cb0-a35ff9b57dff" (UID: "fe0b8634-6037-4df8-9cb0-a35ff9b57dff"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:44:09.211072 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.210983    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-bound-sa-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.211072 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.211025    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-ca-trust-extracted\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.211072 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.211042    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75f8h\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-kube-api-access-75f8h\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.211072 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.211057    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-image-registry-private-configuration\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.211072 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.211073    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-installation-pull-secrets\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.211419 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.211086    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe0b8634-6037-4df8-9cb0-a35ff9b57dff-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:09.773904 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.773861    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs" event={"ID":"fe0b8634-6037-4df8-9cb0-a35ff9b57dff","Type":"ContainerDied","Data":"cff37cd44b48e18ba07c059c75e501341281ce4f7136888aed8eb3d48cbf3c6d"}
May 27 23:44:09.773904 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.773874    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6847498c-gkxfs"
May 27 23:44:09.774430 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.773921    2572 scope.go:117] "RemoveContainer" containerID="bd223bb2f8092b7c68bdb521dceec254af1fe765f397c3f00089141e1f0f1b16"
May 27 23:44:09.795165 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.795138    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6847498c-gkxfs"]
May 27 23:44:09.799800 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:09.799780    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d6847498c-gkxfs"]
May 27 23:44:10.310238 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:10.310204    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:10.310439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:10.310258    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:10.315966 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:10.315938    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:10.781124 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:10.781094    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:44:10.830498 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:10.830459    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6447759fb4-ftgrv"]
May 27 23:44:10.967784 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:10.967752    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:44:11.220033 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:11.220001    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" path="/var/lib/kubelet/pods/fe0b8634-6037-4df8-9cb0-a35ff9b57dff/volumes"
May 27 23:44:15.130624 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:15.130591    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:44:15.131050 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:15.130663    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:44:35.135755 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:35.135716    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:44:35.139494 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:35.139469    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f4857778d-szdtm"
May 27 23:44:35.850775 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:35.850708    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6447759fb4-ftgrv" podUID="6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" containerName="console" containerID="cri-o://8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476" gracePeriod=15
May 27 23:44:36.125051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.125028    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6447759fb4-ftgrv_6fce832b-e0dc-4c8e-9cc9-906010a0d5ea/console/0.log"
May 27 23:44:36.125166 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.125091    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:44:36.228144 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228118    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-oauth-serving-cert\") pod \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") "
May 27 23:44:36.228144 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228149    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-service-ca\") pod \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") "
May 27 23:44:36.228679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228189    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-config\") pod \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") "
May 27 23:44:36.228679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228213    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjftd\" (UniqueName: \"kubernetes.io/projected/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-kube-api-access-pjftd\") pod \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") "
May 27 23:44:36.228679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228297    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-serving-cert\") pod \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") "
May 27 23:44:36.228679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228344    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-oauth-config\") pod \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\" (UID: \"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea\") "
May 27 23:44:36.228679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228602    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" (UID: "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:44:36.228679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228614    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" (UID: "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:44:36.228892 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.228700    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-config" (OuterVolumeSpecName: "console-config") pod "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" (UID: "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:44:36.230641 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.230612    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" (UID: "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:36.230760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.230671    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" (UID: "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:44:36.230760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.230713    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-kube-api-access-pjftd" (OuterVolumeSpecName: "kube-api-access-pjftd") pod "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" (UID: "6fce832b-e0dc-4c8e-9cc9-906010a0d5ea"). InnerVolumeSpecName "kube-api-access-pjftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:44:36.329526 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.329500    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-serving-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:36.329526 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.329522    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-oauth-config\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:36.329663 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.329532    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-oauth-serving-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:36.329663 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.329542    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:36.329663 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.329551    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-console-config\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:36.329663 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.329559    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjftd\" (UniqueName: \"kubernetes.io/projected/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea-kube-api-access-pjftd\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:44:36.854904 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.854874    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6447759fb4-ftgrv_6fce832b-e0dc-4c8e-9cc9-906010a0d5ea/console/0.log"
May 27 23:44:36.855079 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.854921    2572 generic.go:358] "Generic (PLEG): container finished" podID="6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" containerID="8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476" exitCode=2
May 27 23:44:36.855079 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.854978    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6447759fb4-ftgrv"
May 27 23:44:36.855079 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.855005    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447759fb4-ftgrv" event={"ID":"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea","Type":"ContainerDied","Data":"8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476"}
May 27 23:44:36.855079 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.855046    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6447759fb4-ftgrv" event={"ID":"6fce832b-e0dc-4c8e-9cc9-906010a0d5ea","Type":"ContainerDied","Data":"b1adfdeb7c2bc4547bc81c1e1e8f0390ff972b4a9a1bf6a692ec99857e73ffbc"}
May 27 23:44:36.855079 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.855063    2572 scope.go:117] "RemoveContainer" containerID="8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476"
May 27 23:44:36.868564 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.868538    2572 scope.go:117] "RemoveContainer" containerID="8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476"
May 27 23:44:36.868800 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:44:36.868780    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476\": container with ID starting with 8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476 not found: ID does not exist" containerID="8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476"
May 27 23:44:36.868872 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.868812    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476"} err="failed to get container status \"8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476\": rpc error: code = NotFound desc = could not find container \"8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476\": container with ID starting with 8d8010065c509d76d50f0923790ead16cbd11918284bde563b5798cf9adc3476 not found: ID does not exist"
May 27 23:44:36.874589 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.874567    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6447759fb4-ftgrv"]
May 27 23:44:36.878621 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:36.878600    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6447759fb4-ftgrv"]
May 27 23:44:37.219636 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:37.219605    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" path="/var/lib/kubelet/pods/6fce832b-e0dc-4c8e-9cc9-906010a0d5ea/volumes"
May 27 23:44:45.883357 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:45.883299    2572 generic.go:358] "Generic (PLEG): container finished" podID="d5194cbd-67e6-43a3-8698-91ce60449345" containerID="027933029f32ba6cd6c01adea82a79cf8d13f4879755a7796e56749713c25afa" exitCode=0
May 27 23:44:45.883759 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:45.883381    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw" event={"ID":"d5194cbd-67e6-43a3-8698-91ce60449345","Type":"ContainerDied","Data":"027933029f32ba6cd6c01adea82a79cf8d13f4879755a7796e56749713c25afa"}
May 27 23:44:45.883759 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:45.883681    2572 scope.go:117] "RemoveContainer" containerID="027933029f32ba6cd6c01adea82a79cf8d13f4879755a7796e56749713c25afa"
May 27 23:44:46.888167 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:46.888128    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-fhtrw" event={"ID":"d5194cbd-67e6-43a3-8698-91ce60449345","Type":"ContainerStarted","Data":"8b89de533c3bea4e60c8cd39521b23cca67b39ed4f29d30b01567d5d7bb09baf"}
May 27 23:44:50.901151 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:50.901116    2572 generic.go:358] "Generic (PLEG): container finished" podID="4717a5f4-dc35-49e8-a0f1-b2e008417733" containerID="2665c4bd8a216aae089bce7eb963c06097dc090dade06ff45df6c402d2090610" exitCode=0
May 27 23:44:50.901610 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:50.901191    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm" event={"ID":"4717a5f4-dc35-49e8-a0f1-b2e008417733","Type":"ContainerDied","Data":"2665c4bd8a216aae089bce7eb963c06097dc090dade06ff45df6c402d2090610"}
May 27 23:44:50.901610 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:50.901502    2572 scope.go:117] "RemoveContainer" containerID="2665c4bd8a216aae089bce7eb963c06097dc090dade06ff45df6c402d2090610"
May 27 23:44:51.904998 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:44:51.904964    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ddqvm" event={"ID":"4717a5f4-dc35-49e8-a0f1-b2e008417733","Type":"ContainerStarted","Data":"3324f9f5d271925baae14c7356ef5b380991e08f6ed04612caf6e75f3859a294"}
May 27 23:45:03.050344 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:03.050230    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:45:03.052635 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:03.052615    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f-metrics-certs\") pod \"network-metrics-daemon-bp6lz\" (UID: \"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f\") " pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:45:03.119277 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:03.119249    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nr2\""
May 27 23:45:03.126729 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:03.126710    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bp6lz"
May 27 23:45:03.254039 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:03.254000    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bp6lz"]
May 27 23:45:03.258223 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:45:03.258194    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a7eeb9_cd7a_4d9e_b82e_7805d21ddb3f.slice/crio-8d7ae03c7d3974ca4f81fb3963bccea4f1c3d91a023df96d56de66be3d9a9341 WatchSource:0}: Error finding container 8d7ae03c7d3974ca4f81fb3963bccea4f1c3d91a023df96d56de66be3d9a9341: Status 404 returned error can't find the container with id 8d7ae03c7d3974ca4f81fb3963bccea4f1c3d91a023df96d56de66be3d9a9341
May 27 23:45:03.939367 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:03.939331    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bp6lz" event={"ID":"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f","Type":"ContainerStarted","Data":"8d7ae03c7d3974ca4f81fb3963bccea4f1c3d91a023df96d56de66be3d9a9341"}
May 27 23:45:04.943752 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:04.943717    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bp6lz" event={"ID":"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f","Type":"ContainerStarted","Data":"f7fabcf1584780d62f396fbfef62733d479b72b643de0c4fe3e40d1e841e659b"}
May 27 23:45:04.943752 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:04.943754    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bp6lz" event={"ID":"80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f","Type":"ContainerStarted","Data":"cdad2f8cf4f6ed530236eb8413c8df598f58daf4b52c2c12c070ab0eedb6b54b"}
May 27 23:45:04.960122 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:04.960066    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bp6lz" podStartSLOduration=252.743459813 podStartE2EDuration="4m13.960049639s" podCreationTimestamp="2026-05-27 23:40:51 +0000 UTC" firstStartedPulling="2026-05-27 23:45:03.260127062 +0000 UTC m=+252.649630854" lastFinishedPulling="2026-05-27 23:45:04.47671687 +0000 UTC m=+253.866220680" observedRunningTime="2026-05-27 23:45:04.95944262 +0000 UTC m=+254.348946460" watchObservedRunningTime="2026-05-27 23:45:04.960049639 +0000 UTC m=+254.349553453"
May 27 23:45:09.197329 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197274    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc545b668-pkhgk"]
May 27 23:45:09.197811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197679    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" containerName="console"
May 27 23:45:09.197811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197698    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" containerName="console"
May 27 23:45:09.197811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197716    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" containerName="registry"
May 27 23:45:09.197811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197724    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" containerName="registry"
May 27 23:45:09.197811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197795    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe0b8634-6037-4df8-9cb0-a35ff9b57dff" containerName="registry"
May 27 23:45:09.197811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.197809    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="6fce832b-e0dc-4c8e-9cc9-906010a0d5ea" containerName="console"
May 27 23:45:09.199627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.199606    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.212396 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.212375    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc545b668-pkhgk"]
May 27 23:45:09.300990 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.300957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-service-ca\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.300990 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.300991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zmq\" (UniqueName: \"kubernetes.io/projected/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-kube-api-access-l6zmq\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.301174 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.301022    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-trusted-ca-bundle\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.301174 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.301039    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-oauth-serving-cert\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.301174 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.301113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-config\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.301174 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.301148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-oauth-config\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.301301 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.301177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-serving-cert\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.401842 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.401810    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-config\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402041 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.401850    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-oauth-config\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402041 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.401879    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-serving-cert\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402041 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.401906    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-service-ca\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402041 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.401933    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zmq\" (UniqueName: \"kubernetes.io/projected/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-kube-api-access-l6zmq\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402254 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.402079    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-trusted-ca-bundle\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402254 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.402110    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-oauth-serving-cert\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402677 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.402650    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-config\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402846 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.402656    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-service-ca\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.402946 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.402921    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-oauth-serving-cert\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.403105 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.403086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-trusted-ca-bundle\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.404448 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.404430    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-oauth-config\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.404613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.404595    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-serving-cert\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.417570 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.417542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zmq\" (UniqueName: \"kubernetes.io/projected/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-kube-api-access-l6zmq\") pod \"console-dc545b668-pkhgk\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") " pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.508732 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.508637    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:09.636584 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.636551    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc545b668-pkhgk"]
May 27 23:45:09.639260 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:45:09.639233    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5a1aae_ee88_4b02_844e_6e0018d6e11a.slice/crio-26cd8b7cd5f2c29f94c02857c1f2b5132d371d186fc121902285205e7fe798a9 WatchSource:0}: Error finding container 26cd8b7cd5f2c29f94c02857c1f2b5132d371d186fc121902285205e7fe798a9: Status 404 returned error can't find the container with id 26cd8b7cd5f2c29f94c02857c1f2b5132d371d186fc121902285205e7fe798a9
May 27 23:45:09.957756 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.957724    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545b668-pkhgk" event={"ID":"1c5a1aae-ee88-4b02-844e-6e0018d6e11a","Type":"ContainerStarted","Data":"378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773"}
May 27 23:45:09.957756 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.957757    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545b668-pkhgk" event={"ID":"1c5a1aae-ee88-4b02-844e-6e0018d6e11a","Type":"ContainerStarted","Data":"26cd8b7cd5f2c29f94c02857c1f2b5132d371d186fc121902285205e7fe798a9"}
May 27 23:45:09.981628 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:09.981556    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-dc545b668-pkhgk" podStartSLOduration=0.98154403 podStartE2EDuration="981.54403ms" podCreationTimestamp="2026-05-27 23:45:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:45:09.980736516 +0000 UTC m=+259.370240355" watchObservedRunningTime="2026-05-27 23:45:09.98154403 +0000 UTC m=+259.371047848"
May 27 23:45:19.509699 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:19.509657    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:19.510151 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:19.509713    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:19.514901 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:19.514877    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:19.989161 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:19.989133    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:45:20.045516 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:20.045484    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d979df644-t4lqr"]
May 27 23:45:45.064803 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.064759    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7d979df644-t4lqr" podUID="8f213f5d-175c-46aa-8d10-920e8599cd07" containerName="console" containerID="cri-o://f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d" gracePeriod=15
May 27 23:45:45.296856 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.296834    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d979df644-t4lqr_8f213f5d-175c-46aa-8d10-920e8599cd07/console/0.log"
May 27 23:45:45.296966 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.296894    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:45:45.370568 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370490    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kchzv\" (UniqueName: \"kubernetes.io/projected/8f213f5d-175c-46aa-8d10-920e8599cd07-kube-api-access-kchzv\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.370568 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370545    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-oauth-serving-cert\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.370568 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370569    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-serving-cert\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.370760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370598    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-oauth-config\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.370760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370641    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-service-ca\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.370760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370721    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-trusted-ca-bundle\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.370910 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.370805    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-console-config\") pod \"8f213f5d-175c-46aa-8d10-920e8599cd07\" (UID: \"8f213f5d-175c-46aa-8d10-920e8599cd07\") "
May 27 23:45:45.371084 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.371057    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:45:45.371200 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.371056    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:45:45.371270 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.371248    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:45:45.371413 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.371387    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-console-config" (OuterVolumeSpecName: "console-config") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:45:45.372777 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.372745    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:45:45.372862 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.372828    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:45:45.372862 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.372838    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f213f5d-175c-46aa-8d10-920e8599cd07-kube-api-access-kchzv" (OuterVolumeSpecName: "kube-api-access-kchzv") pod "8f213f5d-175c-46aa-8d10-920e8599cd07" (UID: "8f213f5d-175c-46aa-8d10-920e8599cd07"). InnerVolumeSpecName "kube-api-access-kchzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:45:45.472249 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472212    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kchzv\" (UniqueName: \"kubernetes.io/projected/8f213f5d-175c-46aa-8d10-920e8599cd07-kube-api-access-kchzv\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:45.472249 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472246    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-oauth-serving-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:45.472249 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472256    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-serving-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:45.472453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472266    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f213f5d-175c-46aa-8d10-920e8599cd07-console-oauth-config\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:45.472453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472275    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:45.472453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472283    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-trusted-ca-bundle\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:45.472453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:45.472291    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f213f5d-175c-46aa-8d10-920e8599cd07-console-config\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:45:46.063068 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.063040    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d979df644-t4lqr_8f213f5d-175c-46aa-8d10-920e8599cd07/console/0.log"
May 27 23:45:46.063233 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.063080    2572 generic.go:358] "Generic (PLEG): container finished" podID="8f213f5d-175c-46aa-8d10-920e8599cd07" containerID="f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d" exitCode=2
May 27 23:45:46.063233 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.063145    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d979df644-t4lqr"
May 27 23:45:46.063233 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.063179    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d979df644-t4lqr" event={"ID":"8f213f5d-175c-46aa-8d10-920e8599cd07","Type":"ContainerDied","Data":"f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d"}
May 27 23:45:46.063233 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.063218    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d979df644-t4lqr" event={"ID":"8f213f5d-175c-46aa-8d10-920e8599cd07","Type":"ContainerDied","Data":"dc1e815f4a8b8f530f2727340bf5ecffc3fdfb6ac8be709609ecd9b4a6a5a4ef"}
May 27 23:45:46.063397 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.063237    2572 scope.go:117] "RemoveContainer" containerID="f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d"
May 27 23:45:46.071558 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.071366    2572 scope.go:117] "RemoveContainer" containerID="f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d"
May 27 23:45:46.071784 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:45:46.071665    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d\": container with ID starting with f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d not found: ID does not exist" containerID="f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d"
May 27 23:45:46.071784 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.071689    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d"} err="failed to get container status \"f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d\": rpc error: code = NotFound desc = could not find container \"f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d\": container with ID starting with f6e0584bdd03c993e74976fff2b5f423aa57609d3f1fce0e9ff1d92d1a38f05d not found: ID does not exist"
May 27 23:45:46.084958 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.084934    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d979df644-t4lqr"]
May 27 23:45:46.089090 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:46.089068    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d979df644-t4lqr"]
May 27 23:45:47.219439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:47.219406    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f213f5d-175c-46aa-8d10-920e8599cd07" path="/var/lib/kubelet/pods/8f213f5d-175c-46aa-8d10-920e8599cd07/volumes"
May 27 23:45:51.124761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:51.124732    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:45:51.125871 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:51.125847    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:45:51.132373 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:45:51.132355    2572 kubelet.go:1628] "Image garbage collection succeeded"
May 27 23:47:24.114634 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.114592    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"]
May 27 23:47:24.115089 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.114899    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f213f5d-175c-46aa-8d10-920e8599cd07" containerName="console"
May 27 23:47:24.115089 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.114910    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f213f5d-175c-46aa-8d10-920e8599cd07" containerName="console"
May 27 23:47:24.115089 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.114968    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f213f5d-175c-46aa-8d10-920e8599cd07" containerName="console"
May 27 23:47:24.117205 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.117189    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:47:24.119797 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.119765    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 27 23:47:24.119938 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.119885    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 27 23:47:24.120727 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.120707    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-g5k4g\""
May 27 23:47:24.126784 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.126758    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"]
May 27 23:47:24.199767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.199728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgf8c\" (UniqueName: \"kubernetes.io/projected/346eb614-d719-4e25-8f61-e6ac8098e6a9-kube-api-access-vgf8c\") pod \"feast-operator-controller-manager-fcddf6c9f-w5qvr\" (UID: \"346eb614-d719-4e25-8f61-e6ac8098e6a9\") " pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:47:24.300591 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.300551    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vgf8c\" (UniqueName: \"kubernetes.io/projected/346eb614-d719-4e25-8f61-e6ac8098e6a9-kube-api-access-vgf8c\") pod \"feast-operator-controller-manager-fcddf6c9f-w5qvr\" (UID: \"346eb614-d719-4e25-8f61-e6ac8098e6a9\") " pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:47:24.309978 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.309954    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgf8c\" (UniqueName: \"kubernetes.io/projected/346eb614-d719-4e25-8f61-e6ac8098e6a9-kube-api-access-vgf8c\") pod \"feast-operator-controller-manager-fcddf6c9f-w5qvr\" (UID: \"346eb614-d719-4e25-8f61-e6ac8098e6a9\") " pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:47:24.427833 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.427799    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:47:24.553628 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.553591    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"]
May 27 23:47:24.557092 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:47:24.557063    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346eb614_d719_4e25_8f61_e6ac8098e6a9.slice/crio-e9189a9334ec1ed4d477c60f1e83a0435753d7926fe7813790982e9ba7c45b40 WatchSource:0}: Error finding container e9189a9334ec1ed4d477c60f1e83a0435753d7926fe7813790982e9ba7c45b40: Status 404 returned error can't find the container with id e9189a9334ec1ed4d477c60f1e83a0435753d7926fe7813790982e9ba7c45b40
May 27 23:47:24.559053 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:24.559034    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:47:25.327234 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:25.327191    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr" event={"ID":"346eb614-d719-4e25-8f61-e6ac8098e6a9","Type":"ContainerStarted","Data":"e9189a9334ec1ed4d477c60f1e83a0435753d7926fe7813790982e9ba7c45b40"}
May 27 23:47:27.333636 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:27.333601    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr" event={"ID":"346eb614-d719-4e25-8f61-e6ac8098e6a9","Type":"ContainerStarted","Data":"3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca"}
May 27 23:47:27.333992 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:27.333656    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:47:27.350637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:27.350595    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr" podStartSLOduration=1.228652589 podStartE2EDuration="3.350583036s" podCreationTimestamp="2026-05-27 23:47:24 +0000 UTC" firstStartedPulling="2026-05-27 23:47:24.559191328 +0000 UTC m=+393.948695122" lastFinishedPulling="2026-05-27 23:47:26.681121776 +0000 UTC m=+396.070625569" observedRunningTime="2026-05-27 23:47:27.3491021 +0000 UTC m=+396.738605916" watchObservedRunningTime="2026-05-27 23:47:27.350583036 +0000 UTC m=+396.740086850"
May 27 23:47:38.339358 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:47:38.339323    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:48:05.651275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.651199    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"]
May 27 23:48:05.653564 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.653546    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.657111 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.657086    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 27 23:48:05.657357 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.657189    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 27 23:48:05.657471 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.657368    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 27 23:48:05.657471 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.657377    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 27 23:48:05.657471 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.657213    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 27 23:48:05.658282 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.658091    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-kbq78\""
May 27 23:48:05.658282 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.658157    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 27 23:48:05.658455 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.658339    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 27 23:48:05.669716 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.669693    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"]
May 27 23:48:05.706178 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.706146    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f34455e-6321-4084-865e-0c20a8935654-feast-data\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.706359 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.706196    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.706359 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.706250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.706359 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.706279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.706359 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.706297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.706359 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.706339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb579\" (UniqueName: \"kubernetes.io/projected/2f34455e-6321-4084-865e-0c20a8935654-kube-api-access-mb579\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807081 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807052    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807081 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807082    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807101    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mb579\" (UniqueName: \"kubernetes.io/projected/2f34455e-6321-4084-865e-0c20a8935654-kube-api-access-mb579\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807118    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f34455e-6321-4084-865e-0c20a8935654-feast-data\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807152    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807177    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:48:05.807225    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:48:05.807297    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls podName:2f34455e-6321-4084-865e-0c20a8935654 nodeName:}" failed. No retries permitted until 2026-05-27 23:48:06.307275346 +0000 UTC m=+435.696779156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls") pod "feast-simple-feast-setup-768f7d4d54-7d7v2" (UID: "2f34455e-6321-4084-865e-0c20a8935654") : secret "feast-simple-feast-setup-online-tls" not found
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:48:05.807296    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 27 23:48:05.807423 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:48:05.807374    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls podName:2f34455e-6321-4084-865e-0c20a8935654 nodeName:}" failed. No retries permitted until 2026-05-27 23:48:06.307359394 +0000 UTC m=+435.696863186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls") pod "feast-simple-feast-setup-768f7d4d54-7d7v2" (UID: "2f34455e-6321-4084-865e-0c20a8935654") : secret "feast-simple-feast-setup-registry-tls" not found
May 27 23:48:05.807721 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.807598    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f34455e-6321-4084-865e-0c20a8935654-feast-data\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.809780 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.809756    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.809868 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.809796    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:05.818251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:05.818229    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb579\" (UniqueName: \"kubernetes.io/projected/2f34455e-6321-4084-865e-0c20a8935654-kube-api-access-mb579\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:06.311245 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:06.311211    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:06.311426 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:06.311273    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:06.313820 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:06.313796    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:06.313921 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:06.313864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-7d7v2\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:06.577579 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:06.577475    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:06.711012 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:06.710879    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"]
May 27 23:48:06.713338 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:48:06.713293    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f34455e_6321_4084_865e_0c20a8935654.slice/crio-4d5a3ac0ec2a971fe0abcb950509e38e8fe0cec1ca02a40f2c83d5ff0697bbfb WatchSource:0}: Error finding container 4d5a3ac0ec2a971fe0abcb950509e38e8fe0cec1ca02a40f2c83d5ff0697bbfb: Status 404 returned error can't find the container with id 4d5a3ac0ec2a971fe0abcb950509e38e8fe0cec1ca02a40f2c83d5ff0697bbfb
May 27 23:48:07.442666 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:07.442611    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"4d5a3ac0ec2a971fe0abcb950509e38e8fe0cec1ca02a40f2c83d5ff0697bbfb"}
May 27 23:48:37.532982 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:37.532944    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"}
May 27 23:48:41.545097 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:41.545067    2572 generic.go:358] "Generic (PLEG): container finished" podID="2f34455e-6321-4084-865e-0c20a8935654" containerID="6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86" exitCode=0
May 27 23:48:41.545494 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:41.545136    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"}
May 27 23:48:42.550191 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:42.550157    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"}
May 27 23:48:47.566042 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:47.566007    2572 generic.go:358] "Generic (PLEG): container finished" podID="2f34455e-6321-4084-865e-0c20a8935654" containerID="fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b" exitCode=0
May 27 23:48:47.566447 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:47.566100    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"}
May 27 23:48:48.574726 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.574685    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"}
May 27 23:48:48.574726 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.574733    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"}
May 27 23:48:48.575252 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.574748    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"}
May 27 23:48:48.575252 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.574760    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerStarted","Data":"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"}
May 27 23:48:48.578020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.577875    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:48.578020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.577924    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:48.578020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.577939    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:48.578020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.577951    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:48.579961 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.579755    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused"
May 27 23:48:48.579961 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.579818    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused"
May 27 23:48:48.579961 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.579905    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused"
May 27 23:48:48.580356 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.580281    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused"
May 27 23:48:48.597283 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:48.597231    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podStartSLOduration=13.733895624 podStartE2EDuration="43.597208892s" podCreationTimestamp="2026-05-27 23:48:05 +0000 UTC" firstStartedPulling="2026-05-27 23:48:06.715053026 +0000 UTC m=+436.104556819" lastFinishedPulling="2026-05-27 23:48:36.578366296 +0000 UTC m=+465.967870087" observedRunningTime="2026-05-27 23:48:48.596466447 +0000 UTC m=+477.985970262" watchObservedRunningTime="2026-05-27 23:48:48.597208892 +0000 UTC m=+477.986712708"
May 27 23:48:51.578768 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:51.578719    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused"
May 27 23:48:51.579247 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:51.578719    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused"
May 27 23:48:51.579247 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:51.578740    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused"
May 27 23:48:51.579247 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:51.578811    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused"
May 27 23:48:54.579242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.579141    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.579658 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.579456    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.579658 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.579553    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.579999 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.579968    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.580120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.580008    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.580120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.580023    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.580346 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.580301    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.580482 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.580469    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.580530 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.580520    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.584224 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.584201    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.594898 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.594878    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:48:54.597537 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:48:54.597519    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:49:13.050372 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.050335    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f57667c89-mj9fr"]
May 27 23:49:13.076028 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.075997    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f57667c89-mj9fr"]
May 27 23:49:13.076028 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.076036    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164498 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-oauth-serving-cert\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164498 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-console-config\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164693 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/102e367a-1618-4d29-b622-e29d99d617e7-console-oauth-config\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164693 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-service-ca\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164693 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164639    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6knw\" (UniqueName: \"kubernetes.io/projected/102e367a-1618-4d29-b622-e29d99d617e7-kube-api-access-x6knw\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164693 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/102e367a-1618-4d29-b622-e29d99d617e7-console-serving-cert\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.164837 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.164715    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-trusted-ca-bundle\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265258 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265229    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/102e367a-1618-4d29-b622-e29d99d617e7-console-oauth-config\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265258 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265263    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-service-ca\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265281    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x6knw\" (UniqueName: \"kubernetes.io/projected/102e367a-1618-4d29-b622-e29d99d617e7-kube-api-access-x6knw\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265327    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/102e367a-1618-4d29-b622-e29d99d617e7-console-serving-cert\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265359    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-trusted-ca-bundle\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265384    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-oauth-serving-cert\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.265522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.265408    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-console-config\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.266087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.266063    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-service-ca\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.266194 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.266083    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-oauth-serving-cert\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.266194 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.266165    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-console-config\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.266260 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.266243    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/102e367a-1618-4d29-b622-e29d99d617e7-trusted-ca-bundle\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.268441 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.268420    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/102e367a-1618-4d29-b622-e29d99d617e7-console-serving-cert\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.268552 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.268532    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/102e367a-1618-4d29-b622-e29d99d617e7-console-oauth-config\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.273850 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.273828    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6knw\" (UniqueName: \"kubernetes.io/projected/102e367a-1618-4d29-b622-e29d99d617e7-kube-api-access-x6knw\") pod \"console-5f57667c89-mj9fr\" (UID: \"102e367a-1618-4d29-b622-e29d99d617e7\") " pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.384841 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.384767    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:13.510153 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.510127    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f57667c89-mj9fr"]
May 27 23:49:13.512155 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:49:13.512129    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102e367a_1618_4d29_b622_e29d99d617e7.slice/crio-c29dc1a5c4decbeba330b62e23956c0bb5e7c9caa89029f1ad09a6f925fc8bd8 WatchSource:0}: Error finding container c29dc1a5c4decbeba330b62e23956c0bb5e7c9caa89029f1ad09a6f925fc8bd8: Status 404 returned error can't find the container with id c29dc1a5c4decbeba330b62e23956c0bb5e7c9caa89029f1ad09a6f925fc8bd8
May 27 23:49:13.652590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.652497    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f57667c89-mj9fr" event={"ID":"102e367a-1618-4d29-b622-e29d99d617e7","Type":"ContainerStarted","Data":"aed8e3d91f3eb6a856da97c02e6a9a726d7c4dbb47ea52d5f1002a4a839f2cd3"}
May 27 23:49:13.652590 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.652542    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f57667c89-mj9fr" event={"ID":"102e367a-1618-4d29-b622-e29d99d617e7","Type":"ContainerStarted","Data":"c29dc1a5c4decbeba330b62e23956c0bb5e7c9caa89029f1ad09a6f925fc8bd8"}
May 27 23:49:13.672139 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:13.672084    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5f57667c89-mj9fr" podStartSLOduration=0.672068398 podStartE2EDuration="672.068398ms" podCreationTimestamp="2026-05-27 23:49:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:49:13.671889267 +0000 UTC m=+503.061393078" watchObservedRunningTime="2026-05-27 23:49:13.672068398 +0000 UTC m=+503.061572212"
May 27 23:49:23.385694 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:23.385654    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:23.385694 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:23.385704    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:23.390323 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:23.390284    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:23.686046 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:23.686021    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f57667c89-mj9fr"
May 27 23:49:23.734895 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:23.734857    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc545b668-pkhgk"]
May 27 23:49:48.754664 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:48.754610    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-dc545b668-pkhgk" podUID="1c5a1aae-ee88-4b02-844e-6e0018d6e11a" containerName="console" containerID="cri-o://378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773" gracePeriod=15
May 27 23:49:48.995215 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:48.995193    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-dc545b668-pkhgk_1c5a1aae-ee88-4b02-844e-6e0018d6e11a/console/0.log"
May 27 23:49:48.995339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:48.995254    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:49:49.158091 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158060    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-trusted-ca-bundle\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158119    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-oauth-serving-cert\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158167    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-service-ca\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158187    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-oauth-config\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158218    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-serving-cert\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158434 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158268    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6zmq\" (UniqueName: \"kubernetes.io/projected/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-kube-api-access-l6zmq\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158434 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158289    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-config\") pod \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\" (UID: \"1c5a1aae-ee88-4b02-844e-6e0018d6e11a\") "
May 27 23:49:49.158591 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158562    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:49:49.158652 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158572    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:49:49.158819 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158785    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-config" (OuterVolumeSpecName: "console-config") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:49:49.159095 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.158791    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:49:49.160576 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.160546    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:49:49.160720 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.160696    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-kube-api-access-l6zmq" (OuterVolumeSpecName: "kube-api-access-l6zmq") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "kube-api-access-l6zmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:49:49.160778 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.160706    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c5a1aae-ee88-4b02-844e-6e0018d6e11a" (UID: "1c5a1aae-ee88-4b02-844e-6e0018d6e11a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:49:49.259510 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259484    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.259510 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259508    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-oauth-config\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.259669 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259518    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-serving-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.259669 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259529    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zmq\" (UniqueName: \"kubernetes.io/projected/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-kube-api-access-l6zmq\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.259669 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259538    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-console-config\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.259669 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259546    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-trusted-ca-bundle\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.259669 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.259554    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c5a1aae-ee88-4b02-844e-6e0018d6e11a-oauth-serving-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:49:49.753585 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.753559    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-dc545b668-pkhgk_1c5a1aae-ee88-4b02-844e-6e0018d6e11a/console/0.log"
May 27 23:49:49.753751 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.753601    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c5a1aae-ee88-4b02-844e-6e0018d6e11a" containerID="378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773" exitCode=2
May 27 23:49:49.753751 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.753676    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545b668-pkhgk"
May 27 23:49:49.753751 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.753696    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545b668-pkhgk" event={"ID":"1c5a1aae-ee88-4b02-844e-6e0018d6e11a","Type":"ContainerDied","Data":"378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773"}
May 27 23:49:49.753751 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.753737    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545b668-pkhgk" event={"ID":"1c5a1aae-ee88-4b02-844e-6e0018d6e11a","Type":"ContainerDied","Data":"26cd8b7cd5f2c29f94c02857c1f2b5132d371d186fc121902285205e7fe798a9"}
May 27 23:49:49.753877 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.753754    2572 scope.go:117] "RemoveContainer" containerID="378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773"
May 27 23:49:49.763525 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.763231    2572 scope.go:117] "RemoveContainer" containerID="378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773"
May 27 23:49:49.763750 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:49.763595    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773\": container with ID starting with 378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773 not found: ID does not exist" containerID="378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773"
May 27 23:49:49.763750 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.763618    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773"} err="failed to get container status \"378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773\": rpc error: code = NotFound desc = could not find container \"378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773\": container with ID starting with 378f70eef633ff85a3929275833a87c64508759f9caef954f1d81ab24a1fa773 not found: ID does not exist"
May 27 23:49:49.771857 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.771833    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc545b668-pkhgk"]
May 27 23:49:49.775791 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:49.775767    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc545b668-pkhgk"]
May 27 23:49:51.219138 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:51.219107    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c5a1aae-ee88-4b02-844e-6e0018d6e11a" path="/var/lib/kubelet/pods/1c5a1aae-ee88-4b02-844e-6e0018d6e11a/volumes"
May 27 23:49:56.877493 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:56.877453    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"]
May 27 23:49:56.877938 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:56.877885    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="registry" containerID="cri-o://1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0" gracePeriod=30
May 27 23:49:56.878536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:56.878038    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="offline" containerID="cri-o://b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a" gracePeriod=30
May 27 23:49:56.878536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:56.878138    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online" containerID="cri-o://8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e" gracePeriod=30
May 27 23:49:56.878536 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:56.878137    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui" containerID="cri-o://7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199" gracePeriod=30
May 27 23:49:57.149149 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.149063    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"]
May 27 23:49:57.149413 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.149399    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5a1aae-ee88-4b02-844e-6e0018d6e11a" containerName="console"
May 27 23:49:57.149466 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.149416    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5a1aae-ee88-4b02-844e-6e0018d6e11a" containerName="console"
May 27 23:49:57.149508 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.149489    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c5a1aae-ee88-4b02-844e-6e0018d6e11a" containerName="console"
May 27 23:49:57.154345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.154326    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.156941 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.156918    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-mcwv8\""
May 27 23:49:57.163430 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.163406    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"]
May 27 23:49:57.225087 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.225045    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5fj\" (UniqueName: \"kubernetes.io/projected/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-kube-api-access-8s5fj\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.225261 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.225111    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.225261 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.225173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.225261 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.225228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.225416 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.225261    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-feast-data\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.225416 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.225302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326342 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326290    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326534 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326360    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326534 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326378    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-feast-data\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326534 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326400    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326534 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326425    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 27 23:49:57.326534 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326507    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 27 23:49:57.326534 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326525    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls podName:1c046333-0c6f-4a60-b6a4-1b9debcdab5c nodeName:}" failed. No retries permitted until 2026-05-27 23:49:57.826502551 +0000 UTC m=+547.216006343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls") pod "feast-simple-feast-setup-768f7d4d54-2b6px" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c") : secret "feast-simple-feast-setup-ui-tls" not found
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326441    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5fj\" (UniqueName: \"kubernetes.io/projected/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-kube-api-access-8s5fj\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326507    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326569    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls podName:1c046333-0c6f-4a60-b6a4-1b9debcdab5c nodeName:}" failed. No retries permitted until 2026-05-27 23:49:57.826551781 +0000 UTC m=+547.216055605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls") pod "feast-simple-feast-setup-768f7d4d54-2b6px" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c") : secret "feast-simple-feast-setup-registry-tls" not found
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326635    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls podName:1c046333-0c6f-4a60-b6a4-1b9debcdab5c nodeName:}" failed. No retries permitted until 2026-05-27 23:49:57.826622086 +0000 UTC m=+547.216125886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls") pod "feast-simple-feast-setup-768f7d4d54-2b6px" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c") : secret "feast-simple-feast-setup-offline-tls" not found
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326632    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326689    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 27 23:49:57.326808 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:49:57.326723    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls podName:1c046333-0c6f-4a60-b6a4-1b9debcdab5c nodeName:}" failed. No retries permitted until 2026-05-27 23:49:57.826715303 +0000 UTC m=+547.216219095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls") pod "feast-simple-feast-setup-768f7d4d54-2b6px" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c") : secret "feast-simple-feast-setup-online-tls" not found
May 27 23:49:57.327025 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.326830    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-feast-data\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.335375 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.335355    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5fj\" (UniqueName: \"kubernetes.io/projected/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-kube-api-access-8s5fj\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.780664 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.780569    2572 generic.go:358] "Generic (PLEG): container finished" podID="2f34455e-6321-4084-865e-0c20a8935654" containerID="7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199" exitCode=0
May 27 23:49:57.780664 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.780600    2572 generic.go:358] "Generic (PLEG): container finished" podID="2f34455e-6321-4084-865e-0c20a8935654" containerID="8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e" exitCode=0
May 27 23:49:57.780664 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.780638    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"}
May 27 23:49:57.780863 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.780678    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"}
May 27 23:49:57.829988 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.829957    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.830144 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.830015    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.830144 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.830041    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.830144 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.830065    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.832510 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.832488    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.832653 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.832632    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.832699 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.832664    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:57.832699 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:57.832632    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls\") pod \"feast-simple-feast-setup-768f7d4d54-2b6px\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") " pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:58.064332 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:58.064235    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:49:58.195161 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:58.195137    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"]
May 27 23:49:58.197406 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:49:58.197377    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c046333_0c6f_4a60_b6a4_1b9debcdab5c.slice/crio-bcd7798772a5a5dda9e2c65c8f9379a420ac6daf8dc202ff9d3e8531dc60b69a WatchSource:0}: Error finding container bcd7798772a5a5dda9e2c65c8f9379a420ac6daf8dc202ff9d3e8531dc60b69a: Status 404 returned error can't find the container with id bcd7798772a5a5dda9e2c65c8f9379a420ac6daf8dc202ff9d3e8531dc60b69a
May 27 23:49:58.790400 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:58.790360    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"0adcf7c4715b26a73cc824d09b3743f9c4b8c532503aa45c1e07c1a39285b0e0"}
May 27 23:49:58.790400 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:49:58.790402    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"bcd7798772a5a5dda9e2c65c8f9379a420ac6daf8dc202ff9d3e8531dc60b69a"}
May 27 23:50:02.638854 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:02.638823    2572 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c046333_0c6f_4a60_b6a4_1b9debcdab5c.slice/crio-0adcf7c4715b26a73cc824d09b3743f9c4b8c532503aa45c1e07c1a39285b0e0.scope\": RecentStats: unable to find data in memory cache]"
May 27 23:50:02.804158 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:02.804124    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerID="0adcf7c4715b26a73cc824d09b3743f9c4b8c532503aa45c1e07c1a39285b0e0" exitCode=0
May 27 23:50:02.804341 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:02.804201    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"0adcf7c4715b26a73cc824d09b3743f9c4b8c532503aa45c1e07c1a39285b0e0"}
May 27 23:50:03.808519 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:03.808482    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"1933816e93da8274098c1d14930a47eba8b836737ff79b2de77afbce9f425612"}
May 27 23:50:04.580517 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:04.580469    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused"
May 27 23:50:04.595299 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:04.595263    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused"
May 27 23:50:08.824013 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:08.823979    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerID="1933816e93da8274098c1d14930a47eba8b836737ff79b2de77afbce9f425612" exitCode=0
May 27 23:50:08.824501 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:08.824054    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"1933816e93da8274098c1d14930a47eba8b836737ff79b2de77afbce9f425612"}
May 27 23:50:09.831952 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:09.831908    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"83f86765e6496b19e8c07625baf6b2e2ceeccd0bac71b472ae0a1ba175b4316d"}
May 27 23:50:09.831952 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:09.831957    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"51230efd050ba4e3cf54f8fc17a15a84e33e60df6d352eb5573aba9f8c1585a1"}
May 27 23:50:09.832513 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:09.831973    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"5ed2dcd08a0010d5174e03646ce88a478b7845dc98e049de8c7748db79332a14"}
May 27 23:50:09.832513 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:09.831988    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerStarted","Data":"1e5936c39aaae700815353c9076a1f2d1fd0b569837a9b832c071dcb66c4fe9d"}
May 27 23:50:09.859861 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:09.859779    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podStartSLOduration=12.859751162 podStartE2EDuration="12.859751162s" podCreationTimestamp="2026-05-27 23:49:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:50:09.858375104 +0000 UTC m=+559.247878915" watchObservedRunningTime="2026-05-27 23:50:09.859751162 +0000 UTC m=+559.249254977"
May 27 23:50:10.065322 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.065269    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:10.065572 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.065553    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:10.065687 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.065674    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:10.065798 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.065786    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:10.067525 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.067482    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 27 23:50:10.067525 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.067508    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused"
May 27 23:50:10.067711 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.067551    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 27 23:50:10.067711 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:10.067594    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 27 23:50:13.064844 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:13.064787    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 27 23:50:13.065345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:13.064948    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused"
May 27 23:50:13.065345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:13.065026    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 27 23:50:13.065703 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:13.065670    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 27 23:50:14.580297 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:14.580258    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused"
May 27 23:50:14.595066 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:14.595036    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused"
May 27 23:50:16.066061 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.066021    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.066535 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.066174    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.066602 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.066578    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.066875 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.066859    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.066914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.066885    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.066948 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.066942    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.067140 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.067122    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.070637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.070618    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.855641 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.855607    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.855841 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.855645    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.856139 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.856118    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:16.858535 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:16.858508    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:50:24.580397 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:24.580336    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused"
May 27 23:50:24.580958 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:24.580538    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:50:24.595861 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:24.595829    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused"
May 27 23:50:24.595990 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:24.595923    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:50:27.518549 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.518526    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:50:27.582708 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.582677    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls\") pod \"2f34455e-6321-4084-865e-0c20a8935654\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") "
May 27 23:50:27.582836 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.582757    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-offline-tls\") pod \"2f34455e-6321-4084-865e-0c20a8935654\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") "
May 27 23:50:27.582836 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.582784    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls\") pod \"2f34455e-6321-4084-865e-0c20a8935654\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") "
May 27 23:50:27.582836 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.582800    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-ui-tls\") pod \"2f34455e-6321-4084-865e-0c20a8935654\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") "
May 27 23:50:27.582836 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.582821    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f34455e-6321-4084-865e-0c20a8935654-feast-data\") pod \"2f34455e-6321-4084-865e-0c20a8935654\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") "
May 27 23:50:27.583078 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.582836    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb579\" (UniqueName: \"kubernetes.io/projected/2f34455e-6321-4084-865e-0c20a8935654-kube-api-access-mb579\") pod \"2f34455e-6321-4084-865e-0c20a8935654\" (UID: \"2f34455e-6321-4084-865e-0c20a8935654\") "
May 27 23:50:27.583603 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.583569    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f34455e-6321-4084-865e-0c20a8935654-feast-data" (OuterVolumeSpecName: "feast-data") pod "2f34455e-6321-4084-865e-0c20a8935654" (UID: "2f34455e-6321-4084-865e-0c20a8935654"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:50:27.585175 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.585134    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2f34455e-6321-4084-865e-0c20a8935654" (UID: "2f34455e-6321-4084-865e-0c20a8935654"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:50:27.585292 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.585172    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls" (OuterVolumeSpecName: "online-tls") pod "2f34455e-6321-4084-865e-0c20a8935654" (UID: "2f34455e-6321-4084-865e-0c20a8935654"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:50:27.585292 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.585215    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2f34455e-6321-4084-865e-0c20a8935654" (UID: "2f34455e-6321-4084-865e-0c20a8935654"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:50:27.585497 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.585478    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2f34455e-6321-4084-865e-0c20a8935654" (UID: "2f34455e-6321-4084-865e-0c20a8935654"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:50:27.585583 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.585561    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f34455e-6321-4084-865e-0c20a8935654-kube-api-access-mb579" (OuterVolumeSpecName: "kube-api-access-mb579") pod "2f34455e-6321-4084-865e-0c20a8935654" (UID: "2f34455e-6321-4084-865e-0c20a8935654"). InnerVolumeSpecName "kube-api-access-mb579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:50:27.684033 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.684001    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:50:27.684033 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.684030    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:50:27.684033 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.684039    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f34455e-6321-4084-865e-0c20a8935654-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:50:27.684248 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.684048    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb579\" (UniqueName: \"kubernetes.io/projected/2f34455e-6321-4084-865e-0c20a8935654-kube-api-access-mb579\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:50:27.684248 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.684057    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:50:27.684248 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.684066    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f34455e-6321-4084-865e-0c20a8935654-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:50:27.892456 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892417    2572 generic.go:358] "Generic (PLEG): container finished" podID="2f34455e-6321-4084-865e-0c20a8935654" containerID="b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a" exitCode=137
May 27 23:50:27.892456 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892446    2572 generic.go:358] "Generic (PLEG): container finished" podID="2f34455e-6321-4084-865e-0c20a8935654" containerID="1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0" exitCode=137
May 27 23:50:27.892660 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892458    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"}
May 27 23:50:27.892660 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892486    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"
May 27 23:50:27.892660 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892499    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"}
May 27 23:50:27.892660 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892513    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2" event={"ID":"2f34455e-6321-4084-865e-0c20a8935654","Type":"ContainerDied","Data":"4d5a3ac0ec2a971fe0abcb950509e38e8fe0cec1ca02a40f2c83d5ff0697bbfb"}
May 27 23:50:27.892660 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.892528    2572 scope.go:117] "RemoveContainer" containerID="7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"
May 27 23:50:27.901244 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.901229    2572 scope.go:117] "RemoveContainer" containerID="b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"
May 27 23:50:27.908130 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.908103    2572 scope.go:117] "RemoveContainer" containerID="8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"
May 27 23:50:27.913990 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.913968    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"]
May 27 23:50:27.916632 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.916614    2572 scope.go:117] "RemoveContainer" containerID="1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"
May 27 23:50:27.917995 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.917978    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-7d7v2"]
May 27 23:50:27.923434 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.923420    2572 scope.go:117] "RemoveContainer" containerID="fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"
May 27 23:50:27.929949 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.929932    2572 scope.go:117] "RemoveContainer" containerID="6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"
May 27 23:50:27.936397 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.936371    2572 scope.go:117] "RemoveContainer" containerID="7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"
May 27 23:50:27.936614 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:27.936595    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199\": container with ID starting with 7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199 not found: ID does not exist" containerID="7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"
May 27 23:50:27.936679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.936625    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"} err="failed to get container status \"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199\": rpc error: code = NotFound desc = could not find container \"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199\": container with ID starting with 7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199 not found: ID does not exist"
May 27 23:50:27.936679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.936652    2572 scope.go:117] "RemoveContainer" containerID="b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"
May 27 23:50:27.936913 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:27.936897    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a\": container with ID starting with b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a not found: ID does not exist" containerID="b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"
May 27 23:50:27.936951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.936919    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"} err="failed to get container status \"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a\": rpc error: code = NotFound desc = could not find container \"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a\": container with ID starting with b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a not found: ID does not exist"
May 27 23:50:27.936951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.936935    2572 scope.go:117] "RemoveContainer" containerID="8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"
May 27 23:50:27.937152 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:27.937136    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e\": container with ID starting with 8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e not found: ID does not exist" containerID="8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"
May 27 23:50:27.937195 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937159    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"} err="failed to get container status \"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e\": rpc error: code = NotFound desc = could not find container \"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e\": container with ID starting with 8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e not found: ID does not exist"
May 27 23:50:27.937195 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937174    2572 scope.go:117] "RemoveContainer" containerID="1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"
May 27 23:50:27.937394 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:27.937370    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0\": container with ID starting with 1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0 not found: ID does not exist" containerID="1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"
May 27 23:50:27.937445 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937402    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"} err="failed to get container status \"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0\": rpc error: code = NotFound desc = could not find container \"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0\": container with ID starting with 1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0 not found: ID does not exist"
May 27 23:50:27.937445 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937424    2572 scope.go:117] "RemoveContainer" containerID="fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"
May 27 23:50:27.937659 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:27.937643    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b\": container with ID starting with fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b not found: ID does not exist" containerID="fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"
May 27 23:50:27.937706 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937665    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"} err="failed to get container status \"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b\": rpc error: code = NotFound desc = could not find container \"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b\": container with ID starting with fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b not found: ID does not exist"
May 27 23:50:27.937706 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937686    2572 scope.go:117] "RemoveContainer" containerID="6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"
May 27 23:50:27.937911 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:50:27.937891    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86\": container with ID starting with 6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86 not found: ID does not exist" containerID="6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"
May 27 23:50:27.937997 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937913    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"} err="failed to get container status \"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86\": rpc error: code = NotFound desc = could not find container \"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86\": container with ID starting with 6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86 not found: ID does not exist"
May 27 23:50:27.937997 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.937937    2572 scope.go:117] "RemoveContainer" containerID="7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"
May 27 23:50:27.938183 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938166    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199"} err="failed to get container status \"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199\": rpc error: code = NotFound desc = could not find container \"7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199\": container with ID starting with 7629e77e1cd4ad09e931448bf28237eb90fb990a1f20eb2b6685d8dcd429b199 not found: ID does not exist"
May 27 23:50:27.938228 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938184    2572 scope.go:117] "RemoveContainer" containerID="b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"
May 27 23:50:27.938397 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938379    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a"} err="failed to get container status \"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a\": rpc error: code = NotFound desc = could not find container \"b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a\": container with ID starting with b120e063ea0bf6c9a550d81dd527b83cb5bb6b6b6eedb3e92c562dd93f4e892a not found: ID does not exist"
May 27 23:50:27.938450 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938398    2572 scope.go:117] "RemoveContainer" containerID="8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"
May 27 23:50:27.938629 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938614    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e"} err="failed to get container status \"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e\": rpc error: code = NotFound desc = could not find container \"8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e\": container with ID starting with 8f68dbb18e118887e3faf6c22288c152c192f0ccf18bce6a4f472b03cff97f0e not found: ID does not exist"
May 27 23:50:27.938668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938629    2572 scope.go:117] "RemoveContainer" containerID="1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"
May 27 23:50:27.938809 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938792    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0"} err="failed to get container status \"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0\": rpc error: code = NotFound desc = could not find container \"1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0\": container with ID starting with 1495cea83d9d6d74a0a0447a89e60953f7d8d2d412c5048778d8acbf8047ebd0 not found: ID does not exist"
May 27 23:50:27.938849 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938809    2572 scope.go:117] "RemoveContainer" containerID="fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"
May 27 23:50:27.938968 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938953    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b"} err="failed to get container status \"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b\": rpc error: code = NotFound desc = could not find container \"fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b\": container with ID starting with fb3a780c5a2603631e287cdc7131c74ab5953756636565451407497bd4235b6b not found: ID does not exist"
May 27 23:50:27.939018 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.938970    2572 scope.go:117] "RemoveContainer" containerID="6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"
May 27 23:50:27.939183 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:27.939167    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86"} err="failed to get container status \"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86\": rpc error: code = NotFound desc = could not find container \"6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86\": container with ID starting with 6a7f474414618a1879b3e3ab177b119d26679ab88ef76bef4299681525109e86 not found: ID does not exist"
May 27 23:50:29.219805 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:29.219766    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f34455e-6321-4084-865e-0c20a8935654" path="/var/lib/kubelet/pods/2f34455e-6321-4084-865e-0c20a8935654/volumes"
May 27 23:50:51.147741 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:51.147714    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:50:51.148200 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:50:51.147991    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:51:18.714523 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714449    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"]
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714761    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714772    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714783    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="offline"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714788    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="offline"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714799    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="registry"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714805    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="registry"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714812    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714856    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714866    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="feast-apply"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714871    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="feast-apply"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714890    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="feast-init"
May 27 23:51:18.714940 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714899    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="feast-init"
May 27 23:51:18.715347 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714961    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="ui"
May 27 23:51:18.715347 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714974    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="registry"
May 27 23:51:18.715347 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714985    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="online"
May 27 23:51:18.715347 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.714994    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f34455e-6321-4084-865e-0c20a8935654" containerName="offline"
May 27 23:51:18.718437 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.718420    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.721124 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721102    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 27 23:51:18.721237 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721121    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 27 23:51:18.721298 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721232    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 27 23:51:18.721298 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721258    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 27 23:51:18.721298 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721282    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 27 23:51:18.721545 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721284    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-n788f\""
May 27 23:51:18.721545 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721371    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 27 23:51:18.721545 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.721372    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 27 23:51:18.731547 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.731527    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"]
May 27 23:51:18.770933 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.770901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-offline-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.771051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.770945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdk6c\" (UniqueName: \"kubernetes.io/projected/aed41d8d-e267-4497-a207-703292eef2c2-kube-api-access-zdk6c\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.771051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.770967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.771051 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.771009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-online-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.771156 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.771056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aed41d8d-e267-4497-a207-703292eef2c2-registry-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.771156 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.771081    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aed41d8d-e267-4497-a207-703292eef2c2-feast-data\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871588 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.871554    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zdk6c\" (UniqueName: \"kubernetes.io/projected/aed41d8d-e267-4497-a207-703292eef2c2-kube-api-access-zdk6c\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871588 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.871592    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.871621    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-online-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.871665    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aed41d8d-e267-4497-a207-703292eef2c2-registry-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.871689    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aed41d8d-e267-4497-a207-703292eef2c2-feast-data\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.871709    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-offline-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.871817 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:51:18.871711    2572 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 27 23:51:18.871817 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:51:18.871802    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls podName:aed41d8d-e267-4497-a207-703292eef2c2 nodeName:}" failed. No retries permitted until 2026-05-27 23:51:19.371778875 +0000 UTC m=+628.761282672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls") pod "feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" (UID: "aed41d8d-e267-4497-a207-703292eef2c2") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 27 23:51:18.872178 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.872155    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aed41d8d-e267-4497-a207-703292eef2c2-feast-data\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.872444 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.872421    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aed41d8d-e267-4497-a207-703292eef2c2-registry-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.874281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.874263    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-offline-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.874391 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.874321    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-online-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:18.880795 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:18.880773    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdk6c\" (UniqueName: \"kubernetes.io/projected/aed41d8d-e267-4497-a207-703292eef2c2-kube-api-access-zdk6c\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:19.376144 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:19.376112    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:19.378512 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:19.378494    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls\") pod \"feast-simple-feast-remote-setup-794dc6dcc9-zf4rv\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:19.627649 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:19.627575    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:19.766835 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:19.766776    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"]
May 27 23:51:19.767211 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:51:19.767123    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaed41d8d_e267_4497_a207_703292eef2c2.slice/crio-5c885816f4694cd0791de1927f4939804ef9024fd7e251a7810f08cb4cc6b05e WatchSource:0}: Error finding container 5c885816f4694cd0791de1927f4939804ef9024fd7e251a7810f08cb4cc6b05e: Status 404 returned error can't find the container with id 5c885816f4694cd0791de1927f4939804ef9024fd7e251a7810f08cb4cc6b05e
May 27 23:51:20.044023 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:20.043979    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerStarted","Data":"6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec"}
May 27 23:51:20.044211 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:20.044033    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerStarted","Data":"5c885816f4694cd0791de1927f4939804ef9024fd7e251a7810f08cb4cc6b05e"}
May 27 23:51:25.063877 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:25.063840    2572 generic.go:358] "Generic (PLEG): container finished" podID="aed41d8d-e267-4497-a207-703292eef2c2" containerID="6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec" exitCode=0
May 27 23:51:25.064274 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:25.063917    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerDied","Data":"6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec"}
May 27 23:51:26.068934 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:26.068898    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerStarted","Data":"981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd"}
May 27 23:51:31.087552 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:31.087522    2572 generic.go:358] "Generic (PLEG): container finished" podID="aed41d8d-e267-4497-a207-703292eef2c2" containerID="981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd" exitCode=0
May 27 23:51:31.087954 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:31.087576    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerDied","Data":"981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd"}
May 27 23:51:32.094612 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:32.094574    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerStarted","Data":"6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542"}
May 27 23:51:32.094612 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:32.094626    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerStarted","Data":"96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20"}
May 27 23:51:32.095151 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:32.094639    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerStarted","Data":"4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b"}
May 27 23:51:32.121606 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:32.121548    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podStartSLOduration=14.121529645 podStartE2EDuration="14.121529645s" podCreationTimestamp="2026-05-27 23:51:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:51:32.119853426 +0000 UTC m=+641.509357253" watchObservedRunningTime="2026-05-27 23:51:32.121529645 +0000 UTC m=+641.511033464"
May 27 23:51:34.628691 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:34.628646    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:34.629070 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:34.628705    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:34.629070 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:34.628719    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:34.630506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:34.630475    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 27 23:51:34.630506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:34.630458    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused"
May 27 23:51:34.630691 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:34.630484    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 27 23:51:37.628640 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.628605    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:37.629085 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.628737    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:37.629325 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.629278    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:37.629453 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.629335    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:37.629521 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.629495    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:37.629708 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.629689    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:37.632877 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:37.632859    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:38.114184 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:38.114153    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:51:38.117052 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:51:38.117035    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:52:39.917113 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:39.917079    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"]
May 27 23:52:39.917659 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:39.917500    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online" containerID="cri-o://4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b" gracePeriod=30
May 27 23:52:39.917659 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:39.917609    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui" containerID="cri-o://6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542" gracePeriod=30
May 27 23:52:39.917779 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:39.917635    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="offline" containerID="cri-o://96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20" gracePeriod=30
May 27 23:52:40.039088 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:40.039053    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"]
May 27 23:52:40.039668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:40.039584    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui" containerID="cri-o://83f86765e6496b19e8c07625baf6b2e2ceeccd0bac71b472ae0a1ba175b4316d" gracePeriod=30
May 27 23:52:40.039668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:40.039592    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="offline" containerID="cri-o://51230efd050ba4e3cf54f8fc17a15a84e33e60df6d352eb5573aba9f8c1585a1" gracePeriod=30
May 27 23:52:40.039862 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:40.039592    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online" containerID="cri-o://5ed2dcd08a0010d5174e03646ce88a478b7845dc98e049de8c7748db79332a14" gracePeriod=30
May 27 23:52:40.040341 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:40.039530    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="registry" containerID="cri-o://1e5936c39aaae700815353c9076a1f2d1fd0b569837a9b832c071dcb66c4fe9d" gracePeriod=30
May 27 23:52:41.295891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.295857    2572 generic.go:358] "Generic (PLEG): container finished" podID="aed41d8d-e267-4497-a207-703292eef2c2" containerID="6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542" exitCode=0
May 27 23:52:41.295891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.295882    2572 generic.go:358] "Generic (PLEG): container finished" podID="aed41d8d-e267-4497-a207-703292eef2c2" containerID="4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b" exitCode=0
May 27 23:52:41.296344 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.295929    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerDied","Data":"6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542"}
May 27 23:52:41.296344 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.295970    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerDied","Data":"4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b"}
May 27 23:52:41.298480 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.298456    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerID="83f86765e6496b19e8c07625baf6b2e2ceeccd0bac71b472ae0a1ba175b4316d" exitCode=0
May 27 23:52:41.298480 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.298477    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerID="5ed2dcd08a0010d5174e03646ce88a478b7845dc98e049de8c7748db79332a14" exitCode=0
May 27 23:52:41.298628 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.298530    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"83f86765e6496b19e8c07625baf6b2e2ceeccd0bac71b472ae0a1ba175b4316d"}
May 27 23:52:41.298628 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:41.298558    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"5ed2dcd08a0010d5174e03646ce88a478b7845dc98e049de8c7748db79332a14"}
May 27 23:52:46.066817 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:46.066774    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 27 23:52:46.856760 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:46.856716    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 27 23:52:47.629951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:47.629911    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 27 23:52:48.114413 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:48.114368    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 27 23:52:56.067754 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:56.067705    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 27 23:52:56.856698 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:56.856653    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 27 23:52:57.629545 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:57.629494    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 27 23:52:58.114634 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:52:58.114595    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 27 23:53:06.067612 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:06.067568    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 27 23:53:06.068063 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:06.067707    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:53:06.856433 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:06.856371    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 27 23:53:06.856625 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:06.856540    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:53:07.629429 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:07.629385    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 27 23:53:07.629815 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:07.629525    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:53:08.114622 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:08.114581    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 27 23:53:08.114805 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:08.114714    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:53:10.152879 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.152858    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:53:10.251091 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251016    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls\") pod \"aed41d8d-e267-4497-a207-703292eef2c2\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") "
May 27 23:53:10.251235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251103    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdk6c\" (UniqueName: \"kubernetes.io/projected/aed41d8d-e267-4497-a207-703292eef2c2-kube-api-access-zdk6c\") pod \"aed41d8d-e267-4497-a207-703292eef2c2\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") "
May 27 23:53:10.251235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251127    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-offline-tls\") pod \"aed41d8d-e267-4497-a207-703292eef2c2\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") "
May 27 23:53:10.251235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251163    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aed41d8d-e267-4497-a207-703292eef2c2-feast-data\") pod \"aed41d8d-e267-4497-a207-703292eef2c2\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") "
May 27 23:53:10.251235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251193    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-online-tls\") pod \"aed41d8d-e267-4497-a207-703292eef2c2\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") "
May 27 23:53:10.251471 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251272    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aed41d8d-e267-4497-a207-703292eef2c2-registry-tls\") pod \"aed41d8d-e267-4497-a207-703292eef2c2\" (UID: \"aed41d8d-e267-4497-a207-703292eef2c2\") "
May 27 23:53:10.251635 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251605    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed41d8d-e267-4497-a207-703292eef2c2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aed41d8d-e267-4497-a207-703292eef2c2" (UID: "aed41d8d-e267-4497-a207-703292eef2c2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 23:53:10.251928 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.251878    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed41d8d-e267-4497-a207-703292eef2c2-feast-data" (OuterVolumeSpecName: "feast-data") pod "aed41d8d-e267-4497-a207-703292eef2c2" (UID: "aed41d8d-e267-4497-a207-703292eef2c2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:53:10.253446 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.253423    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "aed41d8d-e267-4497-a207-703292eef2c2" (UID: "aed41d8d-e267-4497-a207-703292eef2c2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.253540 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.253464    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-online-tls" (OuterVolumeSpecName: "online-tls") pod "aed41d8d-e267-4497-a207-703292eef2c2" (UID: "aed41d8d-e267-4497-a207-703292eef2c2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.253540 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.253474    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed41d8d-e267-4497-a207-703292eef2c2-kube-api-access-zdk6c" (OuterVolumeSpecName: "kube-api-access-zdk6c") pod "aed41d8d-e267-4497-a207-703292eef2c2" (UID: "aed41d8d-e267-4497-a207-703292eef2c2"). InnerVolumeSpecName "kube-api-access-zdk6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:53:10.253613 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.253558    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "aed41d8d-e267-4497-a207-703292eef2c2" (UID: "aed41d8d-e267-4497-a207-703292eef2c2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.352225 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.352185    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdk6c\" (UniqueName: \"kubernetes.io/projected/aed41d8d-e267-4497-a207-703292eef2c2-kube-api-access-zdk6c\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.352225 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.352224    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.352490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.352240    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aed41d8d-e267-4497-a207-703292eef2c2-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.352490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.352254    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.352490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.352267    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aed41d8d-e267-4497-a207-703292eef2c2-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.352490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.352280    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aed41d8d-e267-4497-a207-703292eef2c2-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.380357 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.380330    2572 generic.go:358] "Generic (PLEG): container finished" podID="aed41d8d-e267-4497-a207-703292eef2c2" containerID="96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20" exitCode=137
May 27 23:53:10.380490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.380393    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"
May 27 23:53:10.380490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.380420    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerDied","Data":"96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20"}
May 27 23:53:10.380490 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.380471    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv" event={"ID":"aed41d8d-e267-4497-a207-703292eef2c2","Type":"ContainerDied","Data":"5c885816f4694cd0791de1927f4939804ef9024fd7e251a7810f08cb4cc6b05e"}
May 27 23:53:10.380614 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.380494    2572 scope.go:117] "RemoveContainer" containerID="6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542"
May 27 23:53:10.383382 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.383360    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerID="51230efd050ba4e3cf54f8fc17a15a84e33e60df6d352eb5573aba9f8c1585a1" exitCode=137
May 27 23:53:10.383382 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.383378    2572 generic.go:358] "Generic (PLEG): container finished" podID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerID="1e5936c39aaae700815353c9076a1f2d1fd0b569837a9b832c071dcb66c4fe9d" exitCode=137
May 27 23:53:10.383489 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.383434    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"51230efd050ba4e3cf54f8fc17a15a84e33e60df6d352eb5573aba9f8c1585a1"}
May 27 23:53:10.383489 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.383465    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"1e5936c39aaae700815353c9076a1f2d1fd0b569837a9b832c071dcb66c4fe9d"}
May 27 23:53:10.391240 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.391201    2572 scope.go:117] "RemoveContainer" containerID="96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20"
May 27 23:53:10.398592 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.398576    2572 scope.go:117] "RemoveContainer" containerID="4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b"
May 27 23:53:10.404134 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.404110    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"]
May 27 23:53:10.405758 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.405743    2572 scope.go:117] "RemoveContainer" containerID="981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd"
May 27 23:53:10.409407 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.409387    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-794dc6dcc9-zf4rv"]
May 27 23:53:10.412835 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.412812    2572 scope.go:117] "RemoveContainer" containerID="6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec"
May 27 23:53:10.419341 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.419301    2572 scope.go:117] "RemoveContainer" containerID="6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542"
May 27 23:53:10.419565 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:53:10.419549    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542\": container with ID starting with 6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542 not found: ID does not exist" containerID="6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542"
May 27 23:53:10.419620 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.419585    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542"} err="failed to get container status \"6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542\": rpc error: code = NotFound desc = could not find container \"6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542\": container with ID starting with 6c0fdff6cde23f3b508993fdebfc36ab92577fa8a99e7099eefbbc84c8324542 not found: ID does not exist"
May 27 23:53:10.419620 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.419604    2572 scope.go:117] "RemoveContainer" containerID="96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20"
May 27 23:53:10.419796 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:53:10.419781    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20\": container with ID starting with 96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20 not found: ID does not exist" containerID="96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20"
May 27 23:53:10.419874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.419802    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20"} err="failed to get container status \"96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20\": rpc error: code = NotFound desc = could not find container \"96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20\": container with ID starting with 96602109200a5bf9ace3d8288b17e10b9a6e9cae7bbacaadf9d7e13260a79a20 not found: ID does not exist"
May 27 23:53:10.419874 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.419818    2572 scope.go:117] "RemoveContainer" containerID="4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b"
May 27 23:53:10.420002 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:53:10.419983    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b\": container with ID starting with 4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b not found: ID does not exist" containerID="4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b"
May 27 23:53:10.420060 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.420011    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b"} err="failed to get container status \"4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b\": rpc error: code = NotFound desc = could not find container \"4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b\": container with ID starting with 4af7a466d1d4c11a94c39bb8f1926b190bec9b87669a9f61e13dcaf91a43297b not found: ID does not exist"
May 27 23:53:10.420060 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.420034    2572 scope.go:117] "RemoveContainer" containerID="981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd"
May 27 23:53:10.420246 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:53:10.420229    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd\": container with ID starting with 981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd not found: ID does not exist" containerID="981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd"
May 27 23:53:10.420285 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.420250    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd"} err="failed to get container status \"981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd\": rpc error: code = NotFound desc = could not find container \"981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd\": container with ID starting with 981cd2dbb69214415bc4fd2107a1b30bf1c8bb8015949197f22edb9bbeac6dfd not found: ID does not exist"
May 27 23:53:10.420285 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.420263    2572 scope.go:117] "RemoveContainer" containerID="6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec"
May 27 23:53:10.420638 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:53:10.420620    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec\": container with ID starting with 6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec not found: ID does not exist" containerID="6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec"
May 27 23:53:10.420700 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.420639    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec"} err="failed to get container status \"6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec\": rpc error: code = NotFound desc = could not find container \"6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec\": container with ID starting with 6d947c7a4fedf7be5bbf5c3bdd2a1bd10ae32263acb1520452f7b24b414bffec not found: ID does not exist"
May 27 23:53:10.689609 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.689586    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:53:10.857439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.857351    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls\") pod \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") "
May 27 23:53:10.857439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.857403    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls\") pod \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") "
May 27 23:53:10.857439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.857429    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls\") pod \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") "
May 27 23:53:10.857722 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.857446    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-feast-data\") pod \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") "
May 27 23:53:10.857722 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.857499    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s5fj\" (UniqueName: \"kubernetes.io/projected/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-kube-api-access-8s5fj\") pod \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") "
May 27 23:53:10.857722 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.857551    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls\") pod \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\" (UID: \"1c046333-0c6f-4a60-b6a4-1b9debcdab5c\") "
May 27 23:53:10.858137 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.858030    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-feast-data" (OuterVolumeSpecName: "feast-data") pod "1c046333-0c6f-4a60-b6a4-1b9debcdab5c" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:53:10.859753 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.859723    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls" (OuterVolumeSpecName: "online-tls") pod "1c046333-0c6f-4a60-b6a4-1b9debcdab5c" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.859852 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.859762    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1c046333-0c6f-4a60-b6a4-1b9debcdab5c" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.859852 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.859804    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1c046333-0c6f-4a60-b6a4-1b9debcdab5c" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.860036 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.860016    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-kube-api-access-8s5fj" (OuterVolumeSpecName: "kube-api-access-8s5fj") pod "1c046333-0c6f-4a60-b6a4-1b9debcdab5c" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c"). InnerVolumeSpecName "kube-api-access-8s5fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:53:10.860036 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.860021    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1c046333-0c6f-4a60-b6a4-1b9debcdab5c" (UID: "1c046333-0c6f-4a60-b6a4-1b9debcdab5c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:53:10.959096 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.959070    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.959096 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.959094    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.959251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.959104    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s5fj\" (UniqueName: \"kubernetes.io/projected/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-kube-api-access-8s5fj\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.959251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.959114    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.959251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.959123    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:10.959251 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:10.959130    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1c046333-0c6f-4a60-b6a4-1b9debcdab5c-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:53:11.219261 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.219232    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed41d8d-e267-4497-a207-703292eef2c2" path="/var/lib/kubelet/pods/aed41d8d-e267-4497-a207-703292eef2c2/volumes"
May 27 23:53:11.388600 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.388566    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px" event={"ID":"1c046333-0c6f-4a60-b6a4-1b9debcdab5c","Type":"ContainerDied","Data":"bcd7798772a5a5dda9e2c65c8f9379a420ac6daf8dc202ff9d3e8531dc60b69a"}
May 27 23:53:11.388779 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.388606    2572 scope.go:117] "RemoveContainer" containerID="83f86765e6496b19e8c07625baf6b2e2ceeccd0bac71b472ae0a1ba175b4316d"
May 27 23:53:11.388779 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.388634    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"
May 27 23:53:11.397096 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.397072    2572 scope.go:117] "RemoveContainer" containerID="51230efd050ba4e3cf54f8fc17a15a84e33e60df6d352eb5573aba9f8c1585a1"
May 27 23:53:11.404197 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.404177    2572 scope.go:117] "RemoveContainer" containerID="5ed2dcd08a0010d5174e03646ce88a478b7845dc98e049de8c7748db79332a14"
May 27 23:53:11.409728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.409707    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"]
May 27 23:53:11.412067 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.412049    2572 scope.go:117] "RemoveContainer" containerID="1e5936c39aaae700815353c9076a1f2d1fd0b569837a9b832c071dcb66c4fe9d"
May 27 23:53:11.412128 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.412079    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-768f7d4d54-2b6px"]
May 27 23:53:11.418930 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.418912    2572 scope.go:117] "RemoveContainer" containerID="1933816e93da8274098c1d14930a47eba8b836737ff79b2de77afbce9f425612"
May 27 23:53:11.425524 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:11.425503    2572 scope.go:117] "RemoveContainer" containerID="0adcf7c4715b26a73cc824d09b3743f9c4b8c532503aa45c1e07c1a39285b0e0"
May 27 23:53:13.219741 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:13.219710    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" path="/var/lib/kubelet/pods/1c046333-0c6f-4a60-b6a4-1b9debcdab5c/volumes"
May 27 23:53:22.812731 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.812696    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-9jdb8"]
May 27 23:53:22.813095 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813074    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online"
May 27 23:53:22.813095 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813092    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813109    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813119    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813130    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="registry"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813139    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="registry"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813149    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="offline"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813156    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="offline"
May 27 23:53:22.813168 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813166    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="feast-init"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813173    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="feast-init"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813184    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813192    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813201    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="feast-init"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813208    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="feast-init"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813218    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="offline"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813247    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="offline"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813260    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813268    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813278    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="feast-apply"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813285    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="feast-apply"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813298    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="feast-apply"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813327    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="feast-apply"
May 27 23:53:22.813410 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813407    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="ui"
May 27 23:53:22.813806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813420    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="registry"
May 27 23:53:22.813806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813428    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="online"
May 27 23:53:22.813806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813438    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="online"
May 27 23:53:22.813806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813448    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="offline"
May 27 23:53:22.813806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813461    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c046333-0c6f-4a60-b6a4-1b9debcdab5c" containerName="offline"
May 27 23:53:22.813806 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.813470    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="aed41d8d-e267-4497-a207-703292eef2c2" containerName="ui"
May 27 23:53:22.818196 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.818172    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:22.821919 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.821894    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 27 23:53:22.822889 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.822867    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 27 23:53:22.823000 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.822891    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 27 23:53:22.823000 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.822874    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-t8q88\""
May 27 23:53:22.827391 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.827369    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-9jdb8"]
May 27 23:53:22.853281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.853255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkjb\" (UniqueName: \"kubernetes.io/projected/5b10b443-83fb-4852-9c00-a479330552b5-kube-api-access-nwkjb\") pod \"postgres-7c7656cc6d-9jdb8\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") " pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:22.853415 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.853326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5b10b443-83fb-4852-9c00-a479330552b5-postgresdata\") pod \"postgres-7c7656cc6d-9jdb8\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") " pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:22.859609 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.859586    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-dlktk"]
May 27 23:53:22.862495 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.862477    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:53:22.871402 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.871367    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-dlktk"]
May 27 23:53:22.954447 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.954421    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkjb\" (UniqueName: \"kubernetes.io/projected/5b10b443-83fb-4852-9c00-a479330552b5-kube-api-access-nwkjb\") pod \"postgres-7c7656cc6d-9jdb8\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") " pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:22.954601 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.954456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kfj\" (UniqueName: \"kubernetes.io/projected/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f-kube-api-access-t8kfj\") pod \"redis-854544c579-dlktk\" (UID: \"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f\") " pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:53:22.954601 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.954512    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5b10b443-83fb-4852-9c00-a479330552b5-postgresdata\") pod \"postgres-7c7656cc6d-9jdb8\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") " pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:22.954837 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.954815    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5b10b443-83fb-4852-9c00-a479330552b5-postgresdata\") pod \"postgres-7c7656cc6d-9jdb8\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") " pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:22.963477 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:22.963451    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkjb\" (UniqueName: \"kubernetes.io/projected/5b10b443-83fb-4852-9c00-a479330552b5-kube-api-access-nwkjb\") pod \"postgres-7c7656cc6d-9jdb8\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") " pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:23.055781 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.055756    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kfj\" (UniqueName: \"kubernetes.io/projected/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f-kube-api-access-t8kfj\") pod \"redis-854544c579-dlktk\" (UID: \"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f\") " pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:53:23.064356 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.064273    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kfj\" (UniqueName: \"kubernetes.io/projected/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f-kube-api-access-t8kfj\") pod \"redis-854544c579-dlktk\" (UID: \"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f\") " pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:53:23.137177 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.137147    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:53:23.172648 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.172186    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:53:23.274547 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.274507    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-9jdb8"]
May 27 23:53:23.277463 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:53:23.277432    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b10b443_83fb_4852_9c00_a479330552b5.slice/crio-d02a581d90d13f3e46a740b099f6671ccfd9725bb9adb46c8700d43960934d42 WatchSource:0}: Error finding container d02a581d90d13f3e46a740b099f6671ccfd9725bb9adb46c8700d43960934d42: Status 404 returned error can't find the container with id d02a581d90d13f3e46a740b099f6671ccfd9725bb9adb46c8700d43960934d42
May 27 23:53:23.279600 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.279582    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:53:23.310936 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.310909    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-dlktk"]
May 27 23:53:23.313776 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:53:23.313736    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda97bf25c_c7c3_46cc_8432_3655a7ed4e1f.slice/crio-e3bc8bcf44cc94d60f7f99c509a457cdefa6167bc44bec0742d8fc2cb4c28394 WatchSource:0}: Error finding container e3bc8bcf44cc94d60f7f99c509a457cdefa6167bc44bec0742d8fc2cb4c28394: Status 404 returned error can't find the container with id e3bc8bcf44cc94d60f7f99c509a457cdefa6167bc44bec0742d8fc2cb4c28394
May 27 23:53:23.424002 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.423961    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-dlktk" event={"ID":"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f","Type":"ContainerStarted","Data":"e3bc8bcf44cc94d60f7f99c509a457cdefa6167bc44bec0742d8fc2cb4c28394"}
May 27 23:53:23.424998 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:23.424973    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8" event={"ID":"5b10b443-83fb-4852-9c00-a479330552b5","Type":"ContainerStarted","Data":"d02a581d90d13f3e46a740b099f6671ccfd9725bb9adb46c8700d43960934d42"}
May 27 23:53:25.435452 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:25.435410    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-dlktk" event={"ID":"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f","Type":"ContainerStarted","Data":"9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab"}
May 27 23:53:25.451230 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:25.451182    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-dlktk" podStartSLOduration=1.6769112229999998 podStartE2EDuration="3.451169155s" podCreationTimestamp="2026-05-27 23:53:22 +0000 UTC" firstStartedPulling="2026-05-27 23:53:23.318082991 +0000 UTC m=+752.707586789" lastFinishedPulling="2026-05-27 23:53:25.092340926 +0000 UTC m=+754.481844721" observedRunningTime="2026-05-27 23:53:25.449562508 +0000 UTC m=+754.839066323" watchObservedRunningTime="2026-05-27 23:53:25.451169155 +0000 UTC m=+754.840672946"
May 27 23:53:29.448179 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:29.448146    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8" event={"ID":"5b10b443-83fb-4852-9c00-a479330552b5","Type":"ContainerStarted","Data":"e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8"}
May 27 23:53:29.465112 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:29.465068    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8" podStartSLOduration=1.410285282 podStartE2EDuration="7.465053402s" podCreationTimestamp="2026-05-27 23:53:22 +0000 UTC" firstStartedPulling="2026-05-27 23:53:23.279865386 +0000 UTC m=+752.669369182" lastFinishedPulling="2026-05-27 23:53:29.334633495 +0000 UTC m=+758.724137302" observedRunningTime="2026-05-27 23:53:29.464224133 +0000 UTC m=+758.853727975" watchObservedRunningTime="2026-05-27 23:53:29.465053402 +0000 UTC m=+758.854557215"
May 27 23:53:33.528094 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.528054    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"]
May 27 23:53:33.534724 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.534705    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.537456 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.537430    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 27 23:53:33.537593 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.537574    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 27 23:53:33.537651 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.537593    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 27 23:53:33.537707 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.537666    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 27 23:53:33.537755 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.537737    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-sr7kj\""
May 27 23:53:33.543668 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.543649    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"]
May 27 23:53:33.640100 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.640069    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-online-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.640212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.640109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-registry-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.640212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.640136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c512241f-e1e3-4c5e-8bf8-6537c92f43de-feast-data\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.640212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.640194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-offline-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.640352 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.640217    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbk6l\" (UniqueName: \"kubernetes.io/projected/c512241f-e1e3-4c5e-8bf8-6537c92f43de-kube-api-access-gbk6l\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.741134 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.741106    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-online-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.741134 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.741139    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-registry-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.741353 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.741160    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c512241f-e1e3-4c5e-8bf8-6537c92f43de-feast-data\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.741353 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.741184    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-offline-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.741353 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.741203    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gbk6l\" (UniqueName: \"kubernetes.io/projected/c512241f-e1e3-4c5e-8bf8-6537c92f43de-kube-api-access-gbk6l\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.741702 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.741669    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c512241f-e1e3-4c5e-8bf8-6537c92f43de-feast-data\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.743810 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.743786    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-online-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.743959 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.743939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-offline-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.744016 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.743939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-registry-tls\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.749564 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.749534    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbk6l\" (UniqueName: \"kubernetes.io/projected/c512241f-e1e3-4c5e-8bf8-6537c92f43de-kube-api-access-gbk6l\") pod \"feast-credit-scoring-58655d8f6-5b2lw\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") " pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.844883 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.844794    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:33.974108 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:33.974076    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"]
May 27 23:53:33.976847 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:53:33.976817    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc512241f_e1e3_4c5e_8bf8_6537c92f43de.slice/crio-2242f5d074f917e54d931b0a078cb1bbbb07063fe679455d2d0cb56202072e90 WatchSource:0}: Error finding container 2242f5d074f917e54d931b0a078cb1bbbb07063fe679455d2d0cb56202072e90: Status 404 returned error can't find the container with id 2242f5d074f917e54d931b0a078cb1bbbb07063fe679455d2d0cb56202072e90
May 27 23:53:34.463648 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:34.463603    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerStarted","Data":"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"}
May 27 23:53:34.463648 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:34.463648    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerStarted","Data":"2242f5d074f917e54d931b0a078cb1bbbb07063fe679455d2d0cb56202072e90"}
May 27 23:53:35.467850 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:35.467812    2572 generic.go:358] "Generic (PLEG): container finished" podID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerID="ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a" exitCode=0
May 27 23:53:35.468212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:35.467897    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerDied","Data":"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"}
May 27 23:53:36.472920 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:36.472886    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerStarted","Data":"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"}
May 27 23:53:42.492226 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:42.492192    2572 generic.go:358] "Generic (PLEG): container finished" podID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerID="c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107" exitCode=0
May 27 23:53:42.492635 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:42.492262    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerDied","Data":"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"}
May 27 23:53:43.498208 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.498176    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerStarted","Data":"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"}
May 27 23:53:43.498208 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.498214    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerStarted","Data":"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"}
May 27 23:53:43.498767 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.498223    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerStarted","Data":"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"}
May 27 23:53:43.528804 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.528738    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podStartSLOduration=10.528718712 podStartE2EDuration="10.528718712s" podCreationTimestamp="2026-05-27 23:53:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:53:43.526176437 +0000 UTC m=+772.915680281" watchObservedRunningTime="2026-05-27 23:53:43.528718712 +0000 UTC m=+772.918222527"
May 27 23:53:43.845212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.845117    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:43.845212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.845160    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:43.845212 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:43.845172    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:45.845326 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:45.845271    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:45.845885 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:45.845861    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:45.845885 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:45.845887    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:45.847099 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:45.847068    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused"
May 27 23:53:45.847222 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:45.847100    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 27 23:53:45.847222 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:45.847110    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 27 23:53:48.845699 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:48.845651    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 27 23:53:48.846596 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:48.846573    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:48.847439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:48.847418    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:48.850826 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:48.850802    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:49.520236 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:49.520206    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:51.845441 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:51.845391    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 27 23:53:54.846581 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:54.846550    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:53:54.847293 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:53:54.847274    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:54:53.884201 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.884165    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"]
May 27 23:54:53.884849 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.884651    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry" containerID="cri-o://660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc" gracePeriod=30
May 27 23:54:53.884849 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.884664    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="offline" containerID="cri-o://7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1" gracePeriod=30
May 27 23:54:53.884849 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.884697    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online" containerID="cri-o://19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5" gracePeriod=30
May 27 23:54:53.902137 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.902110    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-9jdb8"]
May 27 23:54:53.902341 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.902301    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8" podUID="5b10b443-83fb-4852-9c00-a479330552b5" containerName="postgres" containerID="cri-o://e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8" gracePeriod=30
May 27 23:54:53.912166 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.912139    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-dlktk"]
May 27 23:54:53.912401 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:53.912351    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-dlktk" podUID="a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" containerName="redis" containerID="cri-o://9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab" gracePeriod=30
May 27 23:54:54.172174 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.172153    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:54:54.175366 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.175345    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:54:54.290928 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.290889    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8kfj\" (UniqueName: \"kubernetes.io/projected/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f-kube-api-access-t8kfj\") pod \"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f\" (UID: \"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f\") "
May 27 23:54:54.291092 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.291006    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwkjb\" (UniqueName: \"kubernetes.io/projected/5b10b443-83fb-4852-9c00-a479330552b5-kube-api-access-nwkjb\") pod \"5b10b443-83fb-4852-9c00-a479330552b5\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") "
May 27 23:54:54.291092 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.291046    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5b10b443-83fb-4852-9c00-a479330552b5-postgresdata\") pod \"5b10b443-83fb-4852-9c00-a479330552b5\" (UID: \"5b10b443-83fb-4852-9c00-a479330552b5\") "
May 27 23:54:54.293120 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.293090    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b10b443-83fb-4852-9c00-a479330552b5-kube-api-access-nwkjb" (OuterVolumeSpecName: "kube-api-access-nwkjb") pod "5b10b443-83fb-4852-9c00-a479330552b5" (UID: "5b10b443-83fb-4852-9c00-a479330552b5"). InnerVolumeSpecName "kube-api-access-nwkjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:54:54.293256 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.293172    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f-kube-api-access-t8kfj" (OuterVolumeSpecName: "kube-api-access-t8kfj") pod "a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" (UID: "a97bf25c-c7c3-46cc-8432-3655a7ed4e1f"). InnerVolumeSpecName "kube-api-access-t8kfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:54:54.319998 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.319929    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b10b443-83fb-4852-9c00-a479330552b5-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "5b10b443-83fb-4852-9c00-a479330552b5" (UID: "5b10b443-83fb-4852-9c00-a479330552b5"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:54:54.391861 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.391820    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkjb\" (UniqueName: \"kubernetes.io/projected/5b10b443-83fb-4852-9c00-a479330552b5-kube-api-access-nwkjb\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:54:54.391861 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.391849    2572 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/5b10b443-83fb-4852-9c00-a479330552b5-postgresdata\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:54:54.391861 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.391861    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kfj\" (UniqueName: \"kubernetes.io/projected/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f-kube-api-access-t8kfj\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:54:54.712688 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.712656    2572 generic.go:358] "Generic (PLEG): container finished" podID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerID="19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5" exitCode=0
May 27 23:54:54.712859 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.712731    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerDied","Data":"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"}
May 27 23:54:54.713697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.713662    2572 generic.go:358] "Generic (PLEG): container finished" podID="a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" containerID="9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab" exitCode=0
May 27 23:54:54.713800 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.713711    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-dlktk"
May 27 23:54:54.713800 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.713733    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-dlktk" event={"ID":"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f","Type":"ContainerDied","Data":"9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab"}
May 27 23:54:54.713800 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.713761    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-dlktk" event={"ID":"a97bf25c-c7c3-46cc-8432-3655a7ed4e1f","Type":"ContainerDied","Data":"e3bc8bcf44cc94d60f7f99c509a457cdefa6167bc44bec0742d8fc2cb4c28394"}
May 27 23:54:54.713800 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.713782    2572 scope.go:117] "RemoveContainer" containerID="9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab"
May 27 23:54:54.717586 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.717559    2572 generic.go:358] "Generic (PLEG): container finished" podID="5b10b443-83fb-4852-9c00-a479330552b5" containerID="e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8" exitCode=0
May 27 23:54:54.717709 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.717608    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8" event={"ID":"5b10b443-83fb-4852-9c00-a479330552b5","Type":"ContainerDied","Data":"e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8"}
May 27 23:54:54.717709 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.717630    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8" event={"ID":"5b10b443-83fb-4852-9c00-a479330552b5","Type":"ContainerDied","Data":"d02a581d90d13f3e46a740b099f6671ccfd9725bb9adb46c8700d43960934d42"}
May 27 23:54:54.717709 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.717609    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-9jdb8"
May 27 23:54:54.726395 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.726371    2572 scope.go:117] "RemoveContainer" containerID="9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab"
May 27 23:54:54.726741 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:54:54.726676    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab\": container with ID starting with 9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab not found: ID does not exist" containerID="9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab"
May 27 23:54:54.726851 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.726756    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab"} err="failed to get container status \"9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab\": rpc error: code = NotFound desc = could not find container \"9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab\": container with ID starting with 9df6e2c07debdf78a3d0b2d1f86aa1e9d03a23fcf092d80c1acf2c46f91372ab not found: ID does not exist"
May 27 23:54:54.726851 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.726775    2572 scope.go:117] "RemoveContainer" containerID="e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8"
May 27 23:54:54.733953 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.733935    2572 scope.go:117] "RemoveContainer" containerID="e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8"
May 27 23:54:54.734205 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:54:54.734186    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8\": container with ID starting with e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8 not found: ID does not exist" containerID="e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8"
May 27 23:54:54.734275 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.734216    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8"} err="failed to get container status \"e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8\": rpc error: code = NotFound desc = could not find container \"e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8\": container with ID starting with e307fb3914cd6a580adbdd99b7bb5b952ae83e5ef1c196d086d5417d07de87b8 not found: ID does not exist"
May 27 23:54:54.737522 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.737502    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-dlktk"]
May 27 23:54:54.743662 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.743642    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-dlktk"]
May 27 23:54:54.753585 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.753564    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-9jdb8"]
May 27 23:54:54.757478 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:54.757458    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-9jdb8"]
May 27 23:54:55.219190 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:55.219115    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b10b443-83fb-4852-9c00-a479330552b5" path="/var/lib/kubelet/pods/5b10b443-83fb-4852-9c00-a479330552b5/volumes"
May 27 23:54:55.219524 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:55.219474    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" path="/var/lib/kubelet/pods/a97bf25c-c7c3-46cc-8432-3655a7ed4e1f/volumes"
May 27 23:54:59.517889 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:54:59.517852    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 27 23:55:09.517221 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:09.517178    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 27 23:55:19.517182 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:19.517133    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 27 23:55:19.517571 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:19.517279    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:55:24.534993 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.534968    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:55:24.629083 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.629053    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-online-tls\") pod \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") "
May 27 23:55:24.629281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.629123    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-registry-tls\") pod \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") "
May 27 23:55:24.629281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.629148    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbk6l\" (UniqueName: \"kubernetes.io/projected/c512241f-e1e3-4c5e-8bf8-6537c92f43de-kube-api-access-gbk6l\") pod \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") "
May 27 23:55:24.629281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.629179    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-offline-tls\") pod \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") "
May 27 23:55:24.629281 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.629203    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c512241f-e1e3-4c5e-8bf8-6537c92f43de-feast-data\") pod \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\" (UID: \"c512241f-e1e3-4c5e-8bf8-6537c92f43de\") "
May 27 23:55:24.631476 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.631442    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-online-tls" (OuterVolumeSpecName: "online-tls") pod "c512241f-e1e3-4c5e-8bf8-6537c92f43de" (UID: "c512241f-e1e3-4c5e-8bf8-6537c92f43de"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:55:24.631589 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.631448    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c512241f-e1e3-4c5e-8bf8-6537c92f43de-kube-api-access-gbk6l" (OuterVolumeSpecName: "kube-api-access-gbk6l") pod "c512241f-e1e3-4c5e-8bf8-6537c92f43de" (UID: "c512241f-e1e3-4c5e-8bf8-6537c92f43de"). InnerVolumeSpecName "kube-api-access-gbk6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:55:24.631589 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.631505    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c512241f-e1e3-4c5e-8bf8-6537c92f43de" (UID: "c512241f-e1e3-4c5e-8bf8-6537c92f43de"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:55:24.631661 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.631618    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c512241f-e1e3-4c5e-8bf8-6537c92f43de" (UID: "c512241f-e1e3-4c5e-8bf8-6537c92f43de"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:55:24.637517 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.637492    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c512241f-e1e3-4c5e-8bf8-6537c92f43de-feast-data" (OuterVolumeSpecName: "feast-data") pod "c512241f-e1e3-4c5e-8bf8-6537c92f43de" (UID: "c512241f-e1e3-4c5e-8bf8-6537c92f43de"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:55:24.730361 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.730327    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:55:24.730361 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.730358    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbk6l\" (UniqueName: \"kubernetes.io/projected/c512241f-e1e3-4c5e-8bf8-6537c92f43de-kube-api-access-gbk6l\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:55:24.730361 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.730370    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:55:24.730564 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.730379    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c512241f-e1e3-4c5e-8bf8-6537c92f43de-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:55:24.730564 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.730387    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c512241f-e1e3-4c5e-8bf8-6537c92f43de-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:55:24.811506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811477    2572 generic.go:358] "Generic (PLEG): container finished" podID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerID="7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1" exitCode=137
May 27 23:55:24.811506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811500    2572 generic.go:358] "Generic (PLEG): container finished" podID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerID="660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc" exitCode=137
May 27 23:55:24.811705 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811560    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"
May 27 23:55:24.811705 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811557    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerDied","Data":"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"}
May 27 23:55:24.811705 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811670    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerDied","Data":"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"}
May 27 23:55:24.811705 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811681    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw" event={"ID":"c512241f-e1e3-4c5e-8bf8-6537c92f43de","Type":"ContainerDied","Data":"2242f5d074f917e54d931b0a078cb1bbbb07063fe679455d2d0cb56202072e90"}
May 27 23:55:24.811705 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.811695    2572 scope.go:117] "RemoveContainer" containerID="7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"
May 27 23:55:24.819949 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.819894    2572 scope.go:117] "RemoveContainer" containerID="19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"
May 27 23:55:24.827730 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.827712    2572 scope.go:117] "RemoveContainer" containerID="660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"
May 27 23:55:24.835444 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.835335    2572 scope.go:117] "RemoveContainer" containerID="c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"
May 27 23:55:24.835915 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.835885    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"]
May 27 23:55:24.839810 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.839790    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-58655d8f6-5b2lw"]
May 27 23:55:24.843110 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.843095    2572 scope.go:117] "RemoveContainer" containerID="ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"
May 27 23:55:24.849595 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.849579    2572 scope.go:117] "RemoveContainer" containerID="7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"
May 27 23:55:24.849836 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:55:24.849820    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1\": container with ID starting with 7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1 not found: ID does not exist" containerID="7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"
May 27 23:55:24.849876 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.849845    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"} err="failed to get container status \"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1\": rpc error: code = NotFound desc = could not find container \"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1\": container with ID starting with 7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1 not found: ID does not exist"
May 27 23:55:24.849876 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.849864    2572 scope.go:117] "RemoveContainer" containerID="19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"
May 27 23:55:24.850093 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:55:24.850075    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5\": container with ID starting with 19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5 not found: ID does not exist" containerID="19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"
May 27 23:55:24.850142 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850099    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"} err="failed to get container status \"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5\": rpc error: code = NotFound desc = could not find container \"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5\": container with ID starting with 19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5 not found: ID does not exist"
May 27 23:55:24.850142 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850118    2572 scope.go:117] "RemoveContainer" containerID="660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"
May 27 23:55:24.850321 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:55:24.850294    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc\": container with ID starting with 660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc not found: ID does not exist" containerID="660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"
May 27 23:55:24.850364 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850325    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"} err="failed to get container status \"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc\": rpc error: code = NotFound desc = could not find container \"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc\": container with ID starting with 660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc not found: ID does not exist"
May 27 23:55:24.850364 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850338    2572 scope.go:117] "RemoveContainer" containerID="c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"
May 27 23:55:24.850551 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:55:24.850533    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107\": container with ID starting with c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107 not found: ID does not exist" containerID="c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"
May 27 23:55:24.850595 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850553    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"} err="failed to get container status \"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107\": rpc error: code = NotFound desc = could not find container \"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107\": container with ID starting with c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107 not found: ID does not exist"
May 27 23:55:24.850595 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850565    2572 scope.go:117] "RemoveContainer" containerID="ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"
May 27 23:55:24.850802 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:55:24.850786    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a\": container with ID starting with ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a not found: ID does not exist" containerID="ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"
May 27 23:55:24.850853 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850806    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"} err="failed to get container status \"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a\": rpc error: code = NotFound desc = could not find container \"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a\": container with ID starting with ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a not found: ID does not exist"
May 27 23:55:24.850853 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.850821    2572 scope.go:117] "RemoveContainer" containerID="7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"
May 27 23:55:24.851035 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851020    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1"} err="failed to get container status \"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1\": rpc error: code = NotFound desc = could not find container \"7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1\": container with ID starting with 7a313777593ba12eece9f4df255e87581215f268c97adaff55570e88b6a567a1 not found: ID does not exist"
May 27 23:55:24.851035 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851034    2572 scope.go:117] "RemoveContainer" containerID="19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"
May 27 23:55:24.851214 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851198    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5"} err="failed to get container status \"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5\": rpc error: code = NotFound desc = could not find container \"19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5\": container with ID starting with 19eb5e0b9c768100fbc18d80f2f608dd2f7d4d40e5315524b086f2418e04f9c5 not found: ID does not exist"
May 27 23:55:24.851257 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851214    2572 scope.go:117] "RemoveContainer" containerID="660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"
May 27 23:55:24.851495 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851475    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc"} err="failed to get container status \"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc\": rpc error: code = NotFound desc = could not find container \"660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc\": container with ID starting with 660f02209468dcb230f828d43d6403e393a7d14406290d8bacd6932fddef75fc not found: ID does not exist"
May 27 23:55:24.851551 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851496    2572 scope.go:117] "RemoveContainer" containerID="c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"
May 27 23:55:24.851713 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851695    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107"} err="failed to get container status \"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107\": rpc error: code = NotFound desc = could not find container \"c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107\": container with ID starting with c6b92beedda7901710fc7c7b1dbff2f1450329167f51db996666815cd542c107 not found: ID does not exist"
May 27 23:55:24.851761 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851715    2572 scope.go:117] "RemoveContainer" containerID="ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"
May 27 23:55:24.851948 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:24.851930    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a"} err="failed to get container status \"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a\": rpc error: code = NotFound desc = could not find container \"ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a\": container with ID starting with ca0d65c9e017e83e4c92f6f311d0603b96ffc1a09de11472185cc6ce265ae12a not found: ID does not exist"
May 27 23:55:25.225850 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:25.225807    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" path="/var/lib/kubelet/pods/c512241f-e1e3-4c5e-8bf8-6537c92f43de/volumes"
May 27 23:55:51.165492 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:51.165453    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:55:51.166742 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:55:51.166718    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 27 23:56:57.141473 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141442    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-4xdnq"]
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141718    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="feast-init"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141735    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="feast-init"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141749    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" containerName="redis"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141754    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" containerName="redis"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141761    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="feast-apply"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141766    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="feast-apply"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141773    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141779    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141787    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141792    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141798    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="offline"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141802    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="offline"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141809    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b10b443-83fb-4852-9c00-a479330552b5" containerName="postgres"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141814    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b10b443-83fb-4852-9c00-a479330552b5" containerName="postgres"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141868    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="online"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141876    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="offline"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141883    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b10b443-83fb-4852-9c00-a479330552b5" containerName="postgres"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141889    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="c512241f-e1e3-4c5e-8bf8-6537c92f43de" containerName="registry"
May 27 23:56:57.143697 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.141897    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a97bf25c-c7c3-46cc-8432-3655a7ed4e1f" containerName="redis"
May 27 23:56:57.144504 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.144487    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:56:57.147100 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.147076    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 27 23:56:57.147234 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.147082    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 27 23:56:57.147234 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.147176    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-dhqsm\""
May 27 23:56:57.162443 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.162419    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-4xdnq"]
May 27 23:56:57.302179 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.302154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56jw\" (UniqueName: \"kubernetes.io/projected/36dcd851-c6b2-4532-8bf1-dd928b9437ac-kube-api-access-n56jw\") pod \"redis-854544c579-4xdnq\" (UID: \"36dcd851-c6b2-4532-8bf1-dd928b9437ac\") " pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:56:57.403249 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.403172    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n56jw\" (UniqueName: \"kubernetes.io/projected/36dcd851-c6b2-4532-8bf1-dd928b9437ac-kube-api-access-n56jw\") pod \"redis-854544c579-4xdnq\" (UID: \"36dcd851-c6b2-4532-8bf1-dd928b9437ac\") " pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:56:57.412439 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.412414    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56jw\" (UniqueName: \"kubernetes.io/projected/36dcd851-c6b2-4532-8bf1-dd928b9437ac-kube-api-access-n56jw\") pod \"redis-854544c579-4xdnq\" (UID: \"36dcd851-c6b2-4532-8bf1-dd928b9437ac\") " pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:56:57.469092 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.469072    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:56:57.589970 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:57.589937    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-4xdnq"]
May 27 23:56:57.592722 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:56:57.592693    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36dcd851_c6b2_4532_8bf1_dd928b9437ac.slice/crio-200c2ac80d267b3b4b873b274770637c276ebb3a8b67829e82cfb5a2877dfb58 WatchSource:0}: Error finding container 200c2ac80d267b3b4b873b274770637c276ebb3a8b67829e82cfb5a2877dfb58: Status 404 returned error can't find the container with id 200c2ac80d267b3b4b873b274770637c276ebb3a8b67829e82cfb5a2877dfb58
May 27 23:56:58.061953 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:58.061913    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-4xdnq" event={"ID":"36dcd851-c6b2-4532-8bf1-dd928b9437ac","Type":"ContainerStarted","Data":"15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955"}
May 27 23:56:58.062129 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:58.061959    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-4xdnq" event={"ID":"36dcd851-c6b2-4532-8bf1-dd928b9437ac","Type":"ContainerStarted","Data":"200c2ac80d267b3b4b873b274770637c276ebb3a8b67829e82cfb5a2877dfb58"}
May 27 23:56:58.077086 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:56:58.077036    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-4xdnq" podStartSLOduration=1.077022875 podStartE2EDuration="1.077022875s" podCreationTimestamp="2026-05-27 23:56:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:56:58.075595432 +0000 UTC m=+967.465099246" watchObservedRunningTime="2026-05-27 23:56:58.077022875 +0000 UTC m=+967.466526688"
May 27 23:57:02.555510 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.555432    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"]
May 27 23:57:02.558632 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.558610    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.561044 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.561019    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 27 23:57:02.568469 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.568439    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"]
May 27 23:57:02.744038 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.744005    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47dl\" (UniqueName: \"kubernetes.io/projected/1a72730a-dced-4cfe-85a2-79a0db6f94a2-kube-api-access-p47dl\") pod \"postgres-7c7656cc6d-d4ttz\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.744038 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.744037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1a72730a-dced-4cfe-85a2-79a0db6f94a2-postgresdata\") pod \"postgres-7c7656cc6d-d4ttz\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.845073 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.844996    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p47dl\" (UniqueName: \"kubernetes.io/projected/1a72730a-dced-4cfe-85a2-79a0db6f94a2-kube-api-access-p47dl\") pod \"postgres-7c7656cc6d-d4ttz\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.845073 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.845032    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1a72730a-dced-4cfe-85a2-79a0db6f94a2-postgresdata\") pod \"postgres-7c7656cc6d-d4ttz\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.845386 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.845369    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1a72730a-dced-4cfe-85a2-79a0db6f94a2-postgresdata\") pod \"postgres-7c7656cc6d-d4ttz\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.855064 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.855044    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47dl\" (UniqueName: \"kubernetes.io/projected/1a72730a-dced-4cfe-85a2-79a0db6f94a2-kube-api-access-p47dl\") pod \"postgres-7c7656cc6d-d4ttz\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.868679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.868661    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:57:02.992337 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:02.992297    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"]
May 27 23:57:02.994072 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:57:02.994044    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a72730a_dced_4cfe_85a2_79a0db6f94a2.slice/crio-bce6715ab6a8ff71c1f6819190ff7f394b2c246acfe740075aa6b1b9a3c0b970 WatchSource:0}: Error finding container bce6715ab6a8ff71c1f6819190ff7f394b2c246acfe740075aa6b1b9a3c0b970: Status 404 returned error can't find the container with id bce6715ab6a8ff71c1f6819190ff7f394b2c246acfe740075aa6b1b9a3c0b970
May 27 23:57:03.076569 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:03.076529    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz" event={"ID":"1a72730a-dced-4cfe-85a2-79a0db6f94a2","Type":"ContainerStarted","Data":"0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58"}
May 27 23:57:03.076569 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:03.076573    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz" event={"ID":"1a72730a-dced-4cfe-85a2-79a0db6f94a2","Type":"ContainerStarted","Data":"bce6715ab6a8ff71c1f6819190ff7f394b2c246acfe740075aa6b1b9a3c0b970"}
May 27 23:57:03.093716 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:03.093662    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz" podStartSLOduration=1.093648036 podStartE2EDuration="1.093648036s" podCreationTimestamp="2026-05-27 23:57:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:57:03.092250414 +0000 UTC m=+972.481754229" watchObservedRunningTime="2026-05-27 23:57:03.093648036 +0000 UTC m=+972.483151884"
May 27 23:57:08.186645 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.186608    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"]
May 27 23:57:08.190020 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.190002    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.192802 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.192777    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 27 23:57:08.192923 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.192777    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 27 23:57:08.192923 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.192778    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 27 23:57:08.192923 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.192782    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-tnlcl\""
May 27 23:57:08.203891 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.203868    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"]
May 27 23:57:08.282122 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.282088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t764m\" (UniqueName: \"kubernetes.io/projected/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-kube-api-access-t764m\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.282258 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.282133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-feast-data\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.282258 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.282152    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-online-tls\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.282258 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.282240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-registry-tls\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.382673 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.382644    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-feast-data\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.382673 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.382674    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-online-tls\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.382852 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.382725    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-registry-tls\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.382852 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.382745    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t764m\" (UniqueName: \"kubernetes.io/projected/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-kube-api-access-t764m\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.382997 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.382979    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-feast-data\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.385401 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.385375    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-registry-tls\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.385499 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.385429    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-online-tls\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.391841 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.391822    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t764m\" (UniqueName: \"kubernetes.io/projected/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-kube-api-access-t764m\") pod \"feast-credit-scoring-c8c48f94c-6j2jv\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") " pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.498870 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.498803    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:08.629482 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:08.629426    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"]
May 27 23:57:08.632320 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:57:08.632276    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a3764e_b1b4_4472_a2ba_632b51d3cdd7.slice/crio-386dadcd43c8c9efa665be1cbd430ee448ab17a36e8d510aad37b396ea345a2f WatchSource:0}: Error finding container 386dadcd43c8c9efa665be1cbd430ee448ab17a36e8d510aad37b396ea345a2f: Status 404 returned error can't find the container with id 386dadcd43c8c9efa665be1cbd430ee448ab17a36e8d510aad37b396ea345a2f
May 27 23:57:09.100746 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:09.100708    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerStarted","Data":"27a7ffdb8de4e6293cbdec47d4334e093423d1ea820c3d496b475785dcd70d1b"}
May 27 23:57:09.100900 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:09.100752    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerStarted","Data":"386dadcd43c8c9efa665be1cbd430ee448ab17a36e8d510aad37b396ea345a2f"}
May 27 23:57:10.104341 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:10.104293    2572 generic.go:358] "Generic (PLEG): container finished" podID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerID="27a7ffdb8de4e6293cbdec47d4334e093423d1ea820c3d496b475785dcd70d1b" exitCode=0
May 27 23:57:10.104711 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:10.104377    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerDied","Data":"27a7ffdb8de4e6293cbdec47d4334e093423d1ea820c3d496b475785dcd70d1b"}
May 27 23:57:11.109679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:11.109641    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerStarted","Data":"94877f49696af3288b8d8a21f16f128a5e721275fb44d911da560a6b7c35e6ca"}
May 27 23:57:18.133984 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:18.133949    2572 generic.go:358] "Generic (PLEG): container finished" podID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerID="94877f49696af3288b8d8a21f16f128a5e721275fb44d911da560a6b7c35e6ca" exitCode=0
May 27 23:57:18.134372 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:18.133999    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerDied","Data":"94877f49696af3288b8d8a21f16f128a5e721275fb44d911da560a6b7c35e6ca"}
May 27 23:57:19.138987 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:19.138954    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerStarted","Data":"00c59b143ead987ab27f67ab10e57a6ce14f3ebe33f3c165626f9d9f2d936717"}
May 27 23:57:19.139375 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:19.138998    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerStarted","Data":"08e51ed2b4cea215830b50518656eb14143aa93f2c364cf9ab28917417fd8c53"}
May 27 23:57:19.162454 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:19.162401    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podStartSLOduration=11.162384488 podStartE2EDuration="11.162384488s" podCreationTimestamp="2026-05-27 23:57:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:57:19.160022682 +0000 UTC m=+988.549526495" watchObservedRunningTime="2026-05-27 23:57:19.162384488 +0000 UTC m=+988.551888301"
May 27 23:57:20.499135 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:20.499082    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:20.499135 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:20.499135    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:20.500889 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:20.500860    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 27 23:57:20.501029 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:20.500858    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 27 23:57:23.499938 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:23.499886    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 27 23:57:23.499938 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:23.499920    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 27 23:57:26.499582 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:26.499533    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 27 23:57:26.504864 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:26.504839    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:26.505291 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:26.505267    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:26.507962 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:26.507939    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:28.499446 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:28.499409    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:29.500178 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:29.500145    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:30.177857 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:30.177833    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:57:33.997437 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:33.997403    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"]
May 27 23:57:34.002541 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.002518    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.004944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.004918    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 27 23:57:34.004944 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.004933    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 27 23:57:34.005104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.004951    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-kbv4v\""
May 27 23:57:34.005104 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.004943    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 27 23:57:34.012627 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.012605    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"]
May 27 23:57:34.083478 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.083452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.083637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.083536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qj4m\" (UniqueName: \"kubernetes.io/projected/469feb40-c7ab-4efd-aaf3-191e10e0c779-kube-api-access-2qj4m\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.083637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.083577    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-online-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.083637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.083599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/469feb40-c7ab-4efd-aaf3-191e10e0c779-feast-data\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.184852 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.184813    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.185032 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.184876    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qj4m\" (UniqueName: \"kubernetes.io/projected/469feb40-c7ab-4efd-aaf3-191e10e0c779-kube-api-access-2qj4m\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.185032 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.184916    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-online-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.185032 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.184941    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/469feb40-c7ab-4efd-aaf3-191e10e0c779-feast-data\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.185032 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:57:34.184971    2572 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found
May 27 23:57:34.185032 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:57:34.185034    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls podName:469feb40-c7ab-4efd-aaf3-191e10e0c779 nodeName:}" failed. No retries permitted until 2026-05-27 23:57:34.685011877 +0000 UTC m=+1004.074515669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls") pod "feast-driver-ranking-66d9f9496d-dfphq" (UID: "469feb40-c7ab-4efd-aaf3-191e10e0c779") : secret "feast-driver-ranking-registry-tls" not found
May 27 23:57:34.185374 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.185354    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/469feb40-c7ab-4efd-aaf3-191e10e0c779-feast-data\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.187905 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.187885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-online-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.193920 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.193902    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qj4m\" (UniqueName: \"kubernetes.io/projected/469feb40-c7ab-4efd-aaf3-191e10e0c779-kube-api-access-2qj4m\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.689943 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.689907    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.692401 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.692375    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls\") pod \"feast-driver-ranking-66d9f9496d-dfphq\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") " pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:34.911531 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:34.911490    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:35.037679 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:35.037651    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"]
May 27 23:57:35.039797 ip-10-0-133-162 kubenswrapper[2572]: W0527 23:57:35.039770    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469feb40_c7ab_4efd_aaf3_191e10e0c779.slice/crio-b2f8aa79d993991a9e119aed8ac3ae5851d7fbfa4c3241795713205bb90ce037 WatchSource:0}: Error finding container b2f8aa79d993991a9e119aed8ac3ae5851d7fbfa4c3241795713205bb90ce037: Status 404 returned error can't find the container with id b2f8aa79d993991a9e119aed8ac3ae5851d7fbfa4c3241795713205bb90ce037
May 27 23:57:35.191965 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:35.191912    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerStarted","Data":"a5199fb69eb6d27099a8bb98bbd68442355dec24d9efb2d9d3b73bf891ec2fcd"}
May 27 23:57:35.191965 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:35.191967    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerStarted","Data":"b2f8aa79d993991a9e119aed8ac3ae5851d7fbfa4c3241795713205bb90ce037"}
May 27 23:57:40.208142 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:40.208104    2572 generic.go:358] "Generic (PLEG): container finished" podID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerID="a5199fb69eb6d27099a8bb98bbd68442355dec24d9efb2d9d3b73bf891ec2fcd" exitCode=0
May 27 23:57:40.208543 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:40.208178    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerDied","Data":"a5199fb69eb6d27099a8bb98bbd68442355dec24d9efb2d9d3b73bf891ec2fcd"}
May 27 23:57:41.219043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:41.219010    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerStarted","Data":"19b872e3b1e179f28eed5cbc5d393760dbee3e346e43f6569f5b766f2a4e5762"}
May 27 23:57:47.234113 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:47.234015    2572 generic.go:358] "Generic (PLEG): container finished" podID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerID="19b872e3b1e179f28eed5cbc5d393760dbee3e346e43f6569f5b766f2a4e5762" exitCode=0
May 27 23:57:47.234113 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:47.234060    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerDied","Data":"19b872e3b1e179f28eed5cbc5d393760dbee3e346e43f6569f5b766f2a4e5762"}
May 27 23:57:48.239459 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:48.239425    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerStarted","Data":"4e39b0b16f2de40602e6be96d6526dc435a2dcf662d1ec56f637368054d98216"}
May 27 23:57:48.239935 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:48.239467    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerStarted","Data":"7bb1405f5109b010158a45e9377a32bbfcde11332ae81335d2a5099983d66c3a"}
May 27 23:57:48.263217 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:48.263137    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podStartSLOduration=15.263119304 podStartE2EDuration="15.263119304s" podCreationTimestamp="2026-05-27 23:57:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 23:57:48.26163743 +0000 UTC m=+1017.651141243" watchObservedRunningTime="2026-05-27 23:57:48.263119304 +0000 UTC m=+1017.652623118"
May 27 23:57:49.912170 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:49.912128    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:49.912757 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:49.912182    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:49.913809 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:49.913782    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused"
May 27 23:57:49.913926 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:49.913845    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 27 23:57:52.912352 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:52.912289    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused"
May 27 23:57:53.313169 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:53.313092    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:53.313720 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:53.313690    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:53.316264 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:53.316242    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:54.912423 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:54.912379    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:55.911976 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:55.911930    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused"
May 27 23:57:58.912985 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:58.912956    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:57:58.913624 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:57:58.913605    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:59:11.923202 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:11.923164    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"]
May 27 23:59:11.923807 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:11.923500    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr" podUID="346eb614-d719-4e25-8f61-e6ac8098e6a9" containerName="manager" containerID="cri-o://3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca" gracePeriod=10
May 27 23:59:12.205655 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.205632    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:59:12.260345 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.260293    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"]
May 27 23:59:12.260755 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.260721    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry" containerID="cri-o://08e51ed2b4cea215830b50518656eb14143aa93f2c364cf9ab28917417fd8c53" gracePeriod=30
May 27 23:59:12.260882 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.260777    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online" containerID="cri-o://00c59b143ead987ab27f67ab10e57a6ce14f3ebe33f3c165626f9d9f2d936717" gracePeriod=30
May 27 23:59:12.274243 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.274217    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"]
May 27 23:59:12.274531 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.274491    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry" containerID="cri-o://7bb1405f5109b010158a45e9377a32bbfcde11332ae81335d2a5099983d66c3a" gracePeriod=30
May 27 23:59:12.274594 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.274547    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online" containerID="cri-o://4e39b0b16f2de40602e6be96d6526dc435a2dcf662d1ec56f637368054d98216" gracePeriod=30
May 27 23:59:12.299584 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.299561    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgf8c\" (UniqueName: \"kubernetes.io/projected/346eb614-d719-4e25-8f61-e6ac8098e6a9-kube-api-access-vgf8c\") pod \"346eb614-d719-4e25-8f61-e6ac8098e6a9\" (UID: \"346eb614-d719-4e25-8f61-e6ac8098e6a9\") "
May 27 23:59:12.301753 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.301722    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346eb614-d719-4e25-8f61-e6ac8098e6a9-kube-api-access-vgf8c" (OuterVolumeSpecName: "kube-api-access-vgf8c") pod "346eb614-d719-4e25-8f61-e6ac8098e6a9" (UID: "346eb614-d719-4e25-8f61-e6ac8098e6a9"). InnerVolumeSpecName "kube-api-access-vgf8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:59:12.400509 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.400471    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgf8c\" (UniqueName: \"kubernetes.io/projected/346eb614-d719-4e25-8f61-e6ac8098e6a9-kube-api-access-vgf8c\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:12.483893 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.483819    2572 generic.go:358] "Generic (PLEG): container finished" podID="346eb614-d719-4e25-8f61-e6ac8098e6a9" containerID="3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca" exitCode=0
May 27 23:59:12.483893 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.483887    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"
May 27 23:59:12.484058 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.483903    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr" event={"ID":"346eb614-d719-4e25-8f61-e6ac8098e6a9","Type":"ContainerDied","Data":"3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca"}
May 27 23:59:12.484058 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.483942    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr" event={"ID":"346eb614-d719-4e25-8f61-e6ac8098e6a9","Type":"ContainerDied","Data":"e9189a9334ec1ed4d477c60f1e83a0435753d7926fe7813790982e9ba7c45b40"}
May 27 23:59:12.484058 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.483960    2572 scope.go:117] "RemoveContainer" containerID="3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca"
May 27 23:59:12.495381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.494585    2572 scope.go:117] "RemoveContainer" containerID="3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca"
May 27 23:59:12.495381 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:59:12.494916    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca\": container with ID starting with 3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca not found: ID does not exist" containerID="3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca"
May 27 23:59:12.495381 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.494970    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca"} err="failed to get container status \"3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca\": rpc error: code = NotFound desc = could not find container \"3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca\": container with ID starting with 3d356748ba064e340c581253ed6193473111d5fb2c8385f39a47488a909b80ca not found: ID does not exist"
May 27 23:59:12.508637 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.508614    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"]
May 27 23:59:12.512264 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:12.512243    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-fcddf6c9f-w5qvr"]
May 27 23:59:13.219257 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:13.219223    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="346eb614-d719-4e25-8f61-e6ac8098e6a9" path="/var/lib/kubelet/pods/346eb614-d719-4e25-8f61-e6ac8098e6a9/volumes"
May 27 23:59:13.314162 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:13.314067    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 27 23:59:13.488688 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:13.488648    2572 generic.go:358] "Generic (PLEG): container finished" podID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerID="00c59b143ead987ab27f67ab10e57a6ce14f3ebe33f3c165626f9d9f2d936717" exitCode=0
May 27 23:59:13.488850 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:13.488722    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerDied","Data":"00c59b143ead987ab27f67ab10e57a6ce14f3ebe33f3c165626f9d9f2d936717"}
May 27 23:59:13.490785 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:13.490763    2572 generic.go:358] "Generic (PLEG): container finished" podID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerID="4e39b0b16f2de40602e6be96d6526dc435a2dcf662d1ec56f637368054d98216" exitCode=0
May 27 23:59:13.490888 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:13.490835    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerDied","Data":"4e39b0b16f2de40602e6be96d6526dc435a2dcf662d1ec56f637368054d98216"}
May 27 23:59:14.668030 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:14.667995    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"]
May 27 23:59:14.668428 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:14.668198    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz" podUID="1a72730a-dced-4cfe-85a2-79a0db6f94a2" containerName="postgres" containerID="cri-o://0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58" gracePeriod=30
May 27 23:59:14.671870 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:14.671836    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-4xdnq"]
May 27 23:59:14.672043 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:14.672009    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-4xdnq" podUID="36dcd851-c6b2-4532-8bf1-dd928b9437ac" containerName="redis" containerID="cri-o://15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955" gracePeriod=30
May 27 23:59:14.936701 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:14.936680    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:59:14.939785 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:14.939764    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:59:15.029547 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.029521    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1a72730a-dced-4cfe-85a2-79a0db6f94a2-postgresdata\") pod \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") "
May 27 23:59:15.029701 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.029576    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p47dl\" (UniqueName: \"kubernetes.io/projected/1a72730a-dced-4cfe-85a2-79a0db6f94a2-kube-api-access-p47dl\") pod \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\" (UID: \"1a72730a-dced-4cfe-85a2-79a0db6f94a2\") "
May 27 23:59:15.029701 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.029633    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n56jw\" (UniqueName: \"kubernetes.io/projected/36dcd851-c6b2-4532-8bf1-dd928b9437ac-kube-api-access-n56jw\") pod \"36dcd851-c6b2-4532-8bf1-dd928b9437ac\" (UID: \"36dcd851-c6b2-4532-8bf1-dd928b9437ac\") "
May 27 23:59:15.031914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.031886    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a72730a-dced-4cfe-85a2-79a0db6f94a2-kube-api-access-p47dl" (OuterVolumeSpecName: "kube-api-access-p47dl") pod "1a72730a-dced-4cfe-85a2-79a0db6f94a2" (UID: "1a72730a-dced-4cfe-85a2-79a0db6f94a2"). InnerVolumeSpecName "kube-api-access-p47dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:59:15.031914 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.031899    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dcd851-c6b2-4532-8bf1-dd928b9437ac-kube-api-access-n56jw" (OuterVolumeSpecName: "kube-api-access-n56jw") pod "36dcd851-c6b2-4532-8bf1-dd928b9437ac" (UID: "36dcd851-c6b2-4532-8bf1-dd928b9437ac"). InnerVolumeSpecName "kube-api-access-n56jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:59:15.059506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.059476    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a72730a-dced-4cfe-85a2-79a0db6f94a2-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "1a72730a-dced-4cfe-85a2-79a0db6f94a2" (UID: "1a72730a-dced-4cfe-85a2-79a0db6f94a2"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:59:15.130249 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.130227    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56jw\" (UniqueName: \"kubernetes.io/projected/36dcd851-c6b2-4532-8bf1-dd928b9437ac-kube-api-access-n56jw\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:15.130249 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.130248    2572 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1a72730a-dced-4cfe-85a2-79a0db6f94a2-postgresdata\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:15.130415 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.130257    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47dl\" (UniqueName: \"kubernetes.io/projected/1a72730a-dced-4cfe-85a2-79a0db6f94a2-kube-api-access-p47dl\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:15.498618 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.498525    2572 generic.go:358] "Generic (PLEG): container finished" podID="1a72730a-dced-4cfe-85a2-79a0db6f94a2" containerID="0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58" exitCode=0
May 27 23:59:15.498618 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.498573    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"
May 27 23:59:15.498841 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.498613    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz" event={"ID":"1a72730a-dced-4cfe-85a2-79a0db6f94a2","Type":"ContainerDied","Data":"0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58"}
May 27 23:59:15.498841 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.498653    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-d4ttz" event={"ID":"1a72730a-dced-4cfe-85a2-79a0db6f94a2","Type":"ContainerDied","Data":"bce6715ab6a8ff71c1f6819190ff7f394b2c246acfe740075aa6b1b9a3c0b970"}
May 27 23:59:15.498841 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.498671    2572 scope.go:117] "RemoveContainer" containerID="0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58"
May 27 23:59:15.499786 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.499762    2572 generic.go:358] "Generic (PLEG): container finished" podID="36dcd851-c6b2-4532-8bf1-dd928b9437ac" containerID="15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955" exitCode=0
May 27 23:59:15.499900 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.499811    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-4xdnq"
May 27 23:59:15.499900 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.499837    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-4xdnq" event={"ID":"36dcd851-c6b2-4532-8bf1-dd928b9437ac","Type":"ContainerDied","Data":"15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955"}
May 27 23:59:15.499900 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.499862    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-4xdnq" event={"ID":"36dcd851-c6b2-4532-8bf1-dd928b9437ac","Type":"ContainerDied","Data":"200c2ac80d267b3b4b873b274770637c276ebb3a8b67829e82cfb5a2877dfb58"}
May 27 23:59:15.506344 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.506327    2572 scope.go:117] "RemoveContainer" containerID="0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58"
May 27 23:59:15.506601 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:59:15.506583    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58\": container with ID starting with 0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58 not found: ID does not exist" containerID="0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58"
May 27 23:59:15.506664 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.506609    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58"} err="failed to get container status \"0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58\": rpc error: code = NotFound desc = could not find container \"0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58\": container with ID starting with 0f42b4080fcf21d9742cd110be841dacc226688625a32bac4f0c65de149aea58 not found: ID does not exist"
May 27 23:59:15.506664 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.506627    2572 scope.go:117] "RemoveContainer" containerID="15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955"
May 27 23:59:15.513204 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.513187    2572 scope.go:117] "RemoveContainer" containerID="15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955"
May 27 23:59:15.513505 ip-10-0-133-162 kubenswrapper[2572]: E0527 23:59:15.513485    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955\": container with ID starting with 15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955 not found: ID does not exist" containerID="15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955"
May 27 23:59:15.513585 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.513513    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955"} err="failed to get container status \"15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955\": rpc error: code = NotFound desc = could not find container \"15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955\": container with ID starting with 15f9b6963b2759d20d96c4bff075168d88e52ab76e11e57939824c9eb0f91955 not found: ID does not exist"
May 27 23:59:15.515916 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.515895    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-4xdnq"]
May 27 23:59:15.519603 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.519582    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-4xdnq"]
May 27 23:59:15.531929 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.531908    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"]
May 27 23:59:15.536464 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:15.536446    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-d4ttz"]
May 27 23:59:16.505996 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:16.505960    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 27 23:59:17.219145 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:17.219104    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a72730a-dced-4cfe-85a2-79a0db6f94a2" path="/var/lib/kubelet/pods/1a72730a-dced-4cfe-85a2-79a0db6f94a2/volumes"
May 27 23:59:17.219463 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:17.219450    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36dcd851-c6b2-4532-8bf1-dd928b9437ac" path="/var/lib/kubelet/pods/36dcd851-c6b2-4532-8bf1-dd928b9437ac/volumes"
May 27 23:59:23.314474 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:23.314428    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 27 23:59:26.505947 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:26.505899    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 27 23:59:33.313964 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:33.313926    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 27 23:59:33.314385 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:33.314041    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:59:36.505647 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:36.505609    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 27 23:59:36.506029 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:36.505724    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:59:42.577842 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:42.577761    2572 generic.go:358] "Generic (PLEG): container finished" podID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerID="08e51ed2b4cea215830b50518656eb14143aa93f2c364cf9ab28917417fd8c53" exitCode=137
May 27 23:59:42.578165 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:42.577834    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerDied","Data":"08e51ed2b4cea215830b50518656eb14143aa93f2c364cf9ab28917417fd8c53"}
May 27 23:59:42.579680 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:42.579656    2572 generic.go:358] "Generic (PLEG): container finished" podID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerID="7bb1405f5109b010158a45e9377a32bbfcde11332ae81335d2a5099983d66c3a" exitCode=137
May 27 23:59:42.579792 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:42.579716    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerDied","Data":"7bb1405f5109b010158a45e9377a32bbfcde11332ae81335d2a5099983d66c3a"}
May 27 23:59:42.928509 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:42.928489    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:59:42.931382 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:42.931364    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:59:43.020473 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020446    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/469feb40-c7ab-4efd-aaf3-191e10e0c779-feast-data\") pod \"469feb40-c7ab-4efd-aaf3-191e10e0c779\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") "
May 27 23:59:43.020631 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020487    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-online-tls\") pod \"469feb40-c7ab-4efd-aaf3-191e10e0c779\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") "
May 27 23:59:43.020631 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020510    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-feast-data\") pod \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") "
May 27 23:59:43.020631 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020526    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls\") pod \"469feb40-c7ab-4efd-aaf3-191e10e0c779\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") "
May 27 23:59:43.020631 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020547    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qj4m\" (UniqueName: \"kubernetes.io/projected/469feb40-c7ab-4efd-aaf3-191e10e0c779-kube-api-access-2qj4m\") pod \"469feb40-c7ab-4efd-aaf3-191e10e0c779\" (UID: \"469feb40-c7ab-4efd-aaf3-191e10e0c779\") "
May 27 23:59:43.020631 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020589    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t764m\" (UniqueName: \"kubernetes.io/projected/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-kube-api-access-t764m\") pod \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") "
May 27 23:59:43.020631 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020632    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-online-tls\") pod \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") "
May 27 23:59:43.020974 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.020682    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-registry-tls\") pod \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\" (UID: \"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7\") "
May 27 23:59:43.021030 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.021006    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469feb40-c7ab-4efd-aaf3-191e10e0c779-feast-data" (OuterVolumeSpecName: "feast-data") pod "469feb40-c7ab-4efd-aaf3-191e10e0c779" (UID: "469feb40-c7ab-4efd-aaf3-191e10e0c779"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:59:43.022969 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.022930    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469feb40-c7ab-4efd-aaf3-191e10e0c779-kube-api-access-2qj4m" (OuterVolumeSpecName: "kube-api-access-2qj4m") pod "469feb40-c7ab-4efd-aaf3-191e10e0c779" (UID: "469feb40-c7ab-4efd-aaf3-191e10e0c779"). InnerVolumeSpecName "kube-api-access-2qj4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:59:43.023196 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.023171    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-online-tls" (OuterVolumeSpecName: "online-tls") pod "469feb40-c7ab-4efd-aaf3-191e10e0c779" (UID: "469feb40-c7ab-4efd-aaf3-191e10e0c779"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:59:43.023339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.023269    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "469feb40-c7ab-4efd-aaf3-191e10e0c779" (UID: "469feb40-c7ab-4efd-aaf3-191e10e0c779"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:59:43.023577 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.023506    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-kube-api-access-t764m" (OuterVolumeSpecName: "kube-api-access-t764m") pod "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" (UID: "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7"). InnerVolumeSpecName "kube-api-access-t764m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 23:59:43.023577 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.023543    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" (UID: "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:59:43.023703 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.023612    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-online-tls" (OuterVolumeSpecName: "online-tls") pod "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" (UID: "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 23:59:43.030855 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.030834    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-feast-data" (OuterVolumeSpecName: "feast-data") pod "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" (UID: "e7a3764e-b1b4-4472-a2ba-632b51d3cdd7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 23:59:43.121734 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121679    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.121734 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121700    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/469feb40-c7ab-4efd-aaf3-191e10e0c779-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.121734 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121709    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.121734 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121717    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.121734 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121725    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/469feb40-c7ab-4efd-aaf3-191e10e0c779-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.121734 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121735    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qj4m\" (UniqueName: \"kubernetes.io/projected/469feb40-c7ab-4efd-aaf3-191e10e0c779-kube-api-access-2qj4m\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.122073 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121745    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t764m\" (UniqueName: \"kubernetes.io/projected/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-kube-api-access-t764m\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.122073 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.121755    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 27 23:59:43.584478 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.584445    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv" event={"ID":"e7a3764e-b1b4-4472-a2ba-632b51d3cdd7","Type":"ContainerDied","Data":"386dadcd43c8c9efa665be1cbd430ee448ab17a36e8d510aad37b396ea345a2f"}
May 27 23:59:43.584918 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.584498    2572 scope.go:117] "RemoveContainer" containerID="00c59b143ead987ab27f67ab10e57a6ce14f3ebe33f3c165626f9d9f2d936717"
May 27 23:59:43.584918 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.584456    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"
May 27 23:59:43.586728 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.586699    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq" event={"ID":"469feb40-c7ab-4efd-aaf3-191e10e0c779","Type":"ContainerDied","Data":"b2f8aa79d993991a9e119aed8ac3ae5851d7fbfa4c3241795713205bb90ce037"}
May 27 23:59:43.587007 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.586820    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"
May 27 23:59:43.593108 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.593089    2572 scope.go:117] "RemoveContainer" containerID="08e51ed2b4cea215830b50518656eb14143aa93f2c364cf9ab28917417fd8c53"
May 27 23:59:43.600343 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.600327    2572 scope.go:117] "RemoveContainer" containerID="94877f49696af3288b8d8a21f16f128a5e721275fb44d911da560a6b7c35e6ca"
May 27 23:59:43.602707 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.602673    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"]
May 27 23:59:43.606755 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.606734    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-c8c48f94c-6j2jv"]
May 27 23:59:43.607982 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.607966    2572 scope.go:117] "RemoveContainer" containerID="27a7ffdb8de4e6293cbdec47d4334e093423d1ea820c3d496b475785dcd70d1b"
May 27 23:59:43.614525 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.614509    2572 scope.go:117] "RemoveContainer" containerID="4e39b0b16f2de40602e6be96d6526dc435a2dcf662d1ec56f637368054d98216"
May 27 23:59:43.618242 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.618222    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"]
May 27 23:59:43.621318 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.621281    2572 scope.go:117] "RemoveContainer" containerID="7bb1405f5109b010158a45e9377a32bbfcde11332ae81335d2a5099983d66c3a"
May 27 23:59:43.624072 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.624051    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66d9f9496d-dfphq"]
May 27 23:59:43.628368 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.628346    2572 scope.go:117] "RemoveContainer" containerID="19b872e3b1e179f28eed5cbc5d393760dbee3e346e43f6569f5b766f2a4e5762"
May 27 23:59:43.634785 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:43.634769    2572 scope.go:117] "RemoveContainer" containerID="a5199fb69eb6d27099a8bb98bbd68442355dec24d9efb2d9d3b73bf891ec2fcd"
May 27 23:59:45.219910 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:45.219877    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" path="/var/lib/kubelet/pods/469feb40-c7ab-4efd-aaf3-191e10e0c779/volumes"
May 27 23:59:45.220469 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:45.220452    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" path="/var/lib/kubelet/pods/e7a3764e-b1b4-4472-a2ba-632b51d3cdd7/volumes"
May 27 23:59:50.052153 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052116    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"]
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052513    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052531    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052545    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="feast-apply"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052553    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="feast-apply"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052561    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="feast-init"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052570    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="feast-init"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052586    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eb614-d719-4e25-8f61-e6ac8098e6a9" containerName="manager"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052594    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eb614-d719-4e25-8f61-e6ac8098e6a9" containerName="manager"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052605    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052628    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052637    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="feast-apply"
May 27 23:59:50.052639 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052644    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="feast-apply"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052658    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dcd851-c6b2-4532-8bf1-dd928b9437ac" containerName="redis"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052665    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dcd851-c6b2-4532-8bf1-dd928b9437ac" containerName="redis"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052675    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052685    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052699    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a72730a-dced-4cfe-85a2-79a0db6f94a2" containerName="postgres"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052707    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a72730a-dced-4cfe-85a2-79a0db6f94a2" containerName="postgres"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052715    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="feast-init"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052723    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="feast-init"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052801    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="online"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052813    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="346eb614-d719-4e25-8f61-e6ac8098e6a9" containerName="manager"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052824    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="469feb40-c7ab-4efd-aaf3-191e10e0c779" containerName="registry"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052834    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="online"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052846    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7a3764e-b1b4-4472-a2ba-632b51d3cdd7" containerName="registry"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052855    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="36dcd851-c6b2-4532-8bf1-dd928b9437ac" containerName="redis"
May 27 23:59:50.053339 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.052863    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a72730a-dced-4cfe-85a2-79a0db6f94a2" containerName="postgres"
May 27 23:59:50.058644 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.058622    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.061322 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.061288    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7mjgs\""
May 27 23:59:50.061811 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.061791    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 27 23:59:50.061951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.061812    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 27 23:59:50.061951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.061823    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 27 23:59:50.061951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.061791    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 27 23:59:50.061951 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.061829    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 27 23:59:50.062211 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.062198    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 27 23:59:50.069929 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.069908    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"]
May 27 23:59:50.177235 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.177200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.177414 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.177249    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31f13495-6fad-4bfb-905c-82136524e303-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.177414 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.177332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.177414 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.177379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.177414 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.177408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5zp\" (UniqueName: \"kubernetes.io/projected/31f13495-6fad-4bfb-905c-82136524e303-kube-api-access-tl5zp\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.177594 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.177448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.278656 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.278619    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31f13495-6fad-4bfb-905c-82136524e303-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.278839 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.278671    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.278839 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.278716    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.278839 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.278746    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5zp\" (UniqueName: \"kubernetes.io/projected/31f13495-6fad-4bfb-905c-82136524e303-kube-api-access-tl5zp\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.278839 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.278781    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.278839 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.278808    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.279089 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.279053    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31f13495-6fad-4bfb-905c-82136524e303-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.281419 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.281393    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.281506 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.281444    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.281548 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.281507    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.281588 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.281564    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.288379 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.288358    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5zp\" (UniqueName: \"kubernetes.io/projected/31f13495-6fad-4bfb-905c-82136524e303-kube-api-access-tl5zp\") pod \"feast-simple-feast-setup-664fcfcc58-n67nw\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.379920 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.379849    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 27 23:59:50.514353 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.514327    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"]
May 27 23:59:50.520266 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.520240    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 23:59:50.608445 ip-10-0-133-162 kubenswrapper[2572]: I0527 23:59:50.608414    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"438239cd811ea99a6ca9c5bef4b7d52c331fe496fa420f9a6ba24924d8e9a073"}
May 28 00:00:05.655558 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:05.655498    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"fe333647c92aa0da8237c6664a6abbd196823d5f1e6ecbcbc51d63aa55282ba3"}
May 28 00:00:09.667777 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:09.667739    2572 generic.go:358] "Generic (PLEG): container finished" podID="31f13495-6fad-4bfb-905c-82136524e303" containerID="fe333647c92aa0da8237c6664a6abbd196823d5f1e6ecbcbc51d63aa55282ba3" exitCode=0
May 28 00:00:09.668155 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:09.667815    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"fe333647c92aa0da8237c6664a6abbd196823d5f1e6ecbcbc51d63aa55282ba3"}
May 28 00:00:10.673020 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:10.672983    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"e662bb1676a8d912a5981fe1212c33263d0a746147085d849bd33bed1dd3d90a"}
May 28 00:00:15.687875 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:15.687841    2572 generic.go:358] "Generic (PLEG): container finished" podID="31f13495-6fad-4bfb-905c-82136524e303" containerID="e662bb1676a8d912a5981fe1212c33263d0a746147085d849bd33bed1dd3d90a" exitCode=0
May 28 00:00:15.688251 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:15.687914    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"e662bb1676a8d912a5981fe1212c33263d0a746147085d849bd33bed1dd3d90a"}
May 28 00:00:16.694932 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:16.694888    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"4acfa8562560f499fed68ce25f5e4472032da974c0060f910b15d610cb3b9b4b"}
May 28 00:00:16.694932 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:16.694940    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"cff1cb2e308e8b523ea4327b3238fc3b2edd36395dcc225e77c52e9f4a799b98"}
May 28 00:00:16.695512 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:16.694955    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"f0c35fe30e44bcb3f25da8fe6cda168e2c349dfda0735cc6c945a2b0c3c0d995"}
May 28 00:00:16.695512 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:16.694969    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerStarted","Data":"6ddcfd3dd08587e86e7043b6b3bd845f929d8a6ee8f67ec6f24aceeaac1ab4f8"}
May 28 00:00:16.726252 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:16.726187    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podStartSLOduration=12.206671106 podStartE2EDuration="26.726166163s" podCreationTimestamp="2026-05-27 23:59:50 +0000 UTC" firstStartedPulling="2026-05-27 23:59:50.520421459 +0000 UTC m=+1139.909925251" lastFinishedPulling="2026-05-28 00:00:05.039916501 +0000 UTC m=+1154.429420308" observedRunningTime="2026-05-28 00:00:16.723125221 +0000 UTC m=+1166.112629036" watchObservedRunningTime="2026-05-28 00:00:16.726166163 +0000 UTC m=+1166.115669980"
May 28 00:00:17.381007 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.380952    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:17.381007 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.381000    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:17.381007 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.381015    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:17.381354 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.381027    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:17.383215 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.383045    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 00:00:17.383215 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.383090    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 00:00:17.383215 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.383135    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused"
May 28 00:00:17.383215 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:17.383090    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused"
May 28 00:00:20.380735 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.380691    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:20.381259 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.380749    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:20.381259 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.380765    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:20.381259 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.380968    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused"
May 28 00:00:20.381259 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.380968    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused"
May 28 00:00:20.381564 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.381262    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 00:00:20.381564 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.381413    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:20.381697 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:20.381653    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 00:00:23.382049 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.382010    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.382529 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.382264    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.382529 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.382453    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.383171 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.383147    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.383258 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.383217    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.383389 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.383370    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.386710 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.386691    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:23.720550 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:23.720521    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:00:51.189548 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:51.189520    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:00:51.191183 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:00:51.191163    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:01:26.544498 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.544464    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"]
May 28 00:01:26.544942 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.544915    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="registry" containerID="cri-o://6ddcfd3dd08587e86e7043b6b3bd845f929d8a6ee8f67ec6f24aceeaac1ab4f8" gracePeriod=30
May 28 00:01:26.545029 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.544973    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui" containerID="cri-o://4acfa8562560f499fed68ce25f5e4472032da974c0060f910b15d610cb3b9b4b" gracePeriod=30
May 28 00:01:26.545093 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.545012    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="offline" containerID="cri-o://cff1cb2e308e8b523ea4327b3238fc3b2edd36395dcc225e77c52e9f4a799b98" gracePeriod=30
May 28 00:01:26.545093 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.545053    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online" containerID="cri-o://f0c35fe30e44bcb3f25da8fe6cda168e2c349dfda0735cc6c945a2b0c3c0d995" gracePeriod=30
May 28 00:01:26.847756 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.847666    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"]
May 28 00:01:26.851114 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.851086    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:26.853677 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.853652    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qbs8c\""
May 28 00:01:26.863268 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.863241    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"]
May 28 00:01:26.956759 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.956715    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:26.956950 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.956808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:26.956950 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.956868    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:26.956950 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.956906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxtc\" (UniqueName: \"kubernetes.io/projected/261de952-7269-41a1-b2c9-d2e931bd8489-kube-api-access-brxtc\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:26.956950 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.956933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:26.957118 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:26.956957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/261de952-7269-41a1-b2c9-d2e931bd8489-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058155 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058127    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058239 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058183    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058239 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058215    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-brxtc\" (UniqueName: \"kubernetes.io/projected/261de952-7269-41a1-b2c9-d2e931bd8489-kube-api-access-brxtc\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058239 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058233    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058429 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058251    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/261de952-7269-41a1-b2c9-d2e931bd8489-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058429 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:01:27.058260    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 00:01:27.058429 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058285    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.058429 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:01:27.058367    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls podName:261de952-7269-41a1-b2c9-d2e931bd8489 nodeName:}" failed. No retries permitted until 2026-05-28 00:01:27.55834463 +0000 UTC m=+1236.947848435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls") pod "feast-simple-feast-setup-664fcfcc58-js9mv" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489") : secret "feast-simple-feast-setup-online-tls" not found
May 28 00:01:27.058429 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:01:27.058396    2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 00:01:27.058634 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:01:27.058456    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls podName:261de952-7269-41a1-b2c9-d2e931bd8489 nodeName:}" failed. No retries permitted until 2026-05-28 00:01:27.558438336 +0000 UTC m=+1236.947942147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-js9mv" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 00:01:27.058884 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.058848    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/261de952-7269-41a1-b2c9-d2e931bd8489-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.060738 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.060713    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.060836 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.060779    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.070405 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.070382    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxtc\" (UniqueName: \"kubernetes.io/projected/261de952-7269-41a1-b2c9-d2e931bd8489-kube-api-access-brxtc\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.562889 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.562807    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.562889 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.562870    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.565276 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.565253    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.565276 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.565273    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-js9mv\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.760718 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.760689    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:27.895940 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.895906    2572 generic.go:358] "Generic (PLEG): container finished" podID="31f13495-6fad-4bfb-905c-82136524e303" containerID="4acfa8562560f499fed68ce25f5e4472032da974c0060f910b15d610cb3b9b4b" exitCode=0
May 28 00:01:27.895940 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.895934    2572 generic.go:358] "Generic (PLEG): container finished" podID="31f13495-6fad-4bfb-905c-82136524e303" containerID="f0c35fe30e44bcb3f25da8fe6cda168e2c349dfda0735cc6c945a2b0c3c0d995" exitCode=0
May 28 00:01:27.896169 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.895968    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"4acfa8562560f499fed68ce25f5e4472032da974c0060f910b15d610cb3b9b4b"}
May 28 00:01:27.896169 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.896006    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"f0c35fe30e44bcb3f25da8fe6cda168e2c349dfda0735cc6c945a2b0c3c0d995"}
May 28 00:01:27.898900 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:27.898877    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"]
May 28 00:01:27.899453 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:01:27.899433    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261de952_7269_41a1_b2c9_d2e931bd8489.slice/crio-fad08cc68200e875fae2642484ae38704a6e1c735d699aa6f6ebff9d820c1153 WatchSource:0}: Error finding container fad08cc68200e875fae2642484ae38704a6e1c735d699aa6f6ebff9d820c1153: Status 404 returned error can't find the container with id fad08cc68200e875fae2642484ae38704a6e1c735d699aa6f6ebff9d820c1153
May 28 00:01:28.900285 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:28.900248    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"17fac3d3ab40d491cf56b6f659347acea8093acac08b77376138f9a2c6f0ba9e"}
May 28 00:01:28.900285 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:28.900286    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"fad08cc68200e875fae2642484ae38704a6e1c735d699aa6f6ebff9d820c1153"}
May 28 00:01:32.913908 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:32.913875    2572 generic.go:358] "Generic (PLEG): container finished" podID="261de952-7269-41a1-b2c9-d2e931bd8489" containerID="17fac3d3ab40d491cf56b6f659347acea8093acac08b77376138f9a2c6f0ba9e" exitCode=0
May 28 00:01:32.914287 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:32.913928    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"17fac3d3ab40d491cf56b6f659347acea8093acac08b77376138f9a2c6f0ba9e"}
May 28 00:01:33.383370 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:33.383241    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 00:01:33.717982 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:33.717938    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 00:01:33.918553 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:33.918515    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"3993540e687dcb8b1ee3c4f4adf00583dfb141ff973a2a4db40a3fe4fb40c9c0"}
May 28 00:01:38.933773 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:38.933735    2572 generic.go:358] "Generic (PLEG): container finished" podID="261de952-7269-41a1-b2c9-d2e931bd8489" containerID="3993540e687dcb8b1ee3c4f4adf00583dfb141ff973a2a4db40a3fe4fb40c9c0" exitCode=0
May 28 00:01:38.934140 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:38.933810    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"3993540e687dcb8b1ee3c4f4adf00583dfb141ff973a2a4db40a3fe4fb40c9c0"}
May 28 00:01:39.940641 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:39.940599    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"5368ffb34d25b78d1557798979f1d9fc6b030dec3d7befc7176aa7fb40d3a6ae"}
May 28 00:01:39.940641 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:39.940646    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"ed56eb0a3a71b951f45ddb3ee8c8d8558ed228702f104965a285a1b71d7d46c6"}
May 28 00:01:39.941167 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:39.940661    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"4f9c1daa7122cf997bfbc2814a9a5444b8f38750f8c95ccfcd1f596a0fc9e8c1"}
May 28 00:01:39.941167 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:39.940672    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerStarted","Data":"88a05bfde853d62ddde49751f7ffb561f5874047a2fedcc2e77b00a6a7441064"}
May 28 00:01:39.971461 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:39.971406    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podStartSLOduration=13.971372816 podStartE2EDuration="13.971372816s" podCreationTimestamp="2026-05-28 00:01:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:01:39.968553105 +0000 UTC m=+1249.358056919" watchObservedRunningTime="2026-05-28 00:01:39.971372816 +0000 UTC m=+1249.360876631"
May 28 00:01:42.761653 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.761619    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:42.762163 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.762139    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:42.762163 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.762164    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:42.762421 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.762177    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:42.763859 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.763770    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 00:01:42.763859 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.763791    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused"
May 28 00:01:42.763859 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.763828    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 00:01:42.764077 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:42.763853    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 28 00:01:43.383953 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:43.383907    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 00:01:43.718202 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:43.718154    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 00:01:45.762915 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.762877    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763273 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.762977    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763346 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.763270    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763711 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.763686    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763813 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.763719    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763813 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.763733    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763813 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.763788    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.763976 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.763919    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.764114 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.764099    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.767415 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.767398    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.961769 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.961729    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:45.964598 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:45.964577    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:01:53.383103 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:53.383061    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 00:01:53.383686 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:53.383200    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:01:53.718621 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:53.718572    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 00:01:53.718866 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:53.718699    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:01:56.996437 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:56.996397    2572 generic.go:358] "Generic (PLEG): container finished" podID="31f13495-6fad-4bfb-905c-82136524e303" containerID="cff1cb2e308e8b523ea4327b3238fc3b2edd36395dcc225e77c52e9f4a799b98" exitCode=137
May 28 00:01:56.996437 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:56.996432    2572 generic.go:358] "Generic (PLEG): container finished" podID="31f13495-6fad-4bfb-905c-82136524e303" containerID="6ddcfd3dd08587e86e7043b6b3bd845f929d8a6ee8f67ec6f24aceeaac1ab4f8" exitCode=137
May 28 00:01:56.996829 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:56.996475    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"cff1cb2e308e8b523ea4327b3238fc3b2edd36395dcc225e77c52e9f4a799b98"}
May 28 00:01:56.996829 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:56.996511    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"6ddcfd3dd08587e86e7043b6b3bd845f929d8a6ee8f67ec6f24aceeaac1ab4f8"}
May 28 00:01:57.189172 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.189150    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:01:57.295299 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.295224    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-offline-tls\") pod \"31f13495-6fad-4bfb-905c-82136524e303\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") "
May 28 00:01:57.295299 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.295288    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl5zp\" (UniqueName: \"kubernetes.io/projected/31f13495-6fad-4bfb-905c-82136524e303-kube-api-access-tl5zp\") pod \"31f13495-6fad-4bfb-905c-82136524e303\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") "
May 28 00:01:57.295532 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.295335    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-online-tls\") pod \"31f13495-6fad-4bfb-905c-82136524e303\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") "
May 28 00:01:57.295532 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.295356    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-registry-tls\") pod \"31f13495-6fad-4bfb-905c-82136524e303\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") "
May 28 00:01:57.295532 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.295383    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31f13495-6fad-4bfb-905c-82136524e303-feast-data\") pod \"31f13495-6fad-4bfb-905c-82136524e303\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") "
May 28 00:01:57.295532 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.295419    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-ui-tls\") pod \"31f13495-6fad-4bfb-905c-82136524e303\" (UID: \"31f13495-6fad-4bfb-905c-82136524e303\") "
May 28 00:01:57.296423 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.296374    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f13495-6fad-4bfb-905c-82136524e303-feast-data" (OuterVolumeSpecName: "feast-data") pod "31f13495-6fad-4bfb-905c-82136524e303" (UID: "31f13495-6fad-4bfb-905c-82136524e303"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:01:57.297849 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.297818    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "31f13495-6fad-4bfb-905c-82136524e303" (UID: "31f13495-6fad-4bfb-905c-82136524e303"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:01:57.297955 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.297881    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-online-tls" (OuterVolumeSpecName: "online-tls") pod "31f13495-6fad-4bfb-905c-82136524e303" (UID: "31f13495-6fad-4bfb-905c-82136524e303"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:01:57.297955 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.297902    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f13495-6fad-4bfb-905c-82136524e303-kube-api-access-tl5zp" (OuterVolumeSpecName: "kube-api-access-tl5zp") pod "31f13495-6fad-4bfb-905c-82136524e303" (UID: "31f13495-6fad-4bfb-905c-82136524e303"). InnerVolumeSpecName "kube-api-access-tl5zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:01:57.298104 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.298087    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "31f13495-6fad-4bfb-905c-82136524e303" (UID: "31f13495-6fad-4bfb-905c-82136524e303"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:01:57.298196 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.298174    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "31f13495-6fad-4bfb-905c-82136524e303" (UID: "31f13495-6fad-4bfb-905c-82136524e303"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:01:57.396736 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.396696    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:01:57.396736 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.396728    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:01:57.396736 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.396737    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31f13495-6fad-4bfb-905c-82136524e303-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:01:57.396736 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.396745    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:01:57.396998 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.396757    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31f13495-6fad-4bfb-905c-82136524e303-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:01:57.396998 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:57.396770    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5zp\" (UniqueName: \"kubernetes.io/projected/31f13495-6fad-4bfb-905c-82136524e303-kube-api-access-tl5zp\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:01:58.001760 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.001722    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw" event={"ID":"31f13495-6fad-4bfb-905c-82136524e303","Type":"ContainerDied","Data":"438239cd811ea99a6ca9c5bef4b7d52c331fe496fa420f9a6ba24924d8e9a073"}
May 28 00:01:58.002195 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.001776    2572 scope.go:117] "RemoveContainer" containerID="4acfa8562560f499fed68ce25f5e4472032da974c0060f910b15d610cb3b9b4b"
May 28 00:01:58.002195 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.001783    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"
May 28 00:01:58.011616 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.011592    2572 scope.go:117] "RemoveContainer" containerID="cff1cb2e308e8b523ea4327b3238fc3b2edd36395dcc225e77c52e9f4a799b98"
May 28 00:01:58.018875 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.018856    2572 scope.go:117] "RemoveContainer" containerID="f0c35fe30e44bcb3f25da8fe6cda168e2c349dfda0735cc6c945a2b0c3c0d995"
May 28 00:01:58.025664 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.025633    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"]
May 28 00:01:58.027466 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.027441    2572 scope.go:117] "RemoveContainer" containerID="6ddcfd3dd08587e86e7043b6b3bd845f929d8a6ee8f67ec6f24aceeaac1ab4f8"
May 28 00:01:58.028082 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.028056    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n67nw"]
May 28 00:01:58.034649 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.034627    2572 scope.go:117] "RemoveContainer" containerID="e662bb1676a8d912a5981fe1212c33263d0a746147085d849bd33bed1dd3d90a"
May 28 00:01:58.041117 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:58.041101    2572 scope.go:117] "RemoveContainer" containerID="fe333647c92aa0da8237c6664a6abbd196823d5f1e6ecbcbc51d63aa55282ba3"
May 28 00:01:59.219122 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:01:59.219081    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f13495-6fad-4bfb-905c-82136524e303" path="/var/lib/kubelet/pods/31f13495-6fad-4bfb-905c-82136524e303/volumes"
May 28 00:02:48.378141 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378103    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"]
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378537    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="offline"
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378553    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="offline"
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378569    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="feast-apply"
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378578    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="feast-apply"
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378592    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui"
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378601    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui"
May 28 00:02:48.378622 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378624    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="registry"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378632    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="registry"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378642    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="feast-init"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378650    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="feast-init"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378662    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378671    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378729    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="registry"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378741    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="online"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378753    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="offline"
May 28 00:02:48.378843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.378763    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="31f13495-6fad-4bfb-905c-82136524e303" containerName="ui"
May 28 00:02:48.381243 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.381226    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.383623 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.383593    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 00:02:48.383752 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.383649    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-65qvm\""
May 28 00:02:48.383752 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.383653    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 00:02:48.383752 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.383709    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 00:02:48.383953 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.383934    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 00:02:48.383953 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.383949    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 00:02:48.384451 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.384435    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 00:02:48.390096 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.390072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8gj\" (UniqueName: \"kubernetes.io/projected/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-kube-api-access-hs8gj\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.390212 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.390116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.390212 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.390147    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.390212 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.390192    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.390401 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.390255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.390401 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.390288    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.392106 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.392084    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"]
May 28 00:02:48.490674 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.490630    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.490674 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.490675    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.490716    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8gj\" (UniqueName: \"kubernetes.io/projected/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-kube-api-access-hs8gj\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.490745    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.490769    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.490806    2572 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.490825    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.490878    2572 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 00:02:48.490935 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.490938    2572 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:02:48.491230 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.490882    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls podName:4ace6bee-91c4-4a59-a7e1-feb8b8014ab4 nodeName:}" failed. No retries permitted until 2026-05-28 00:02:48.990857493 +0000 UTC m=+1318.380361286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls") pod "feast-simple-feast-remote-setup-5cff94746b-htx8v" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 00:02:48.491230 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.490991    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls podName:4ace6bee-91c4-4a59-a7e1-feb8b8014ab4 nodeName:}" failed. No retries permitted until 2026-05-28 00:02:48.990969387 +0000 UTC m=+1318.380473185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-htx8v" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 00:02:48.491230 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.491010    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls podName:4ace6bee-91c4-4a59-a7e1-feb8b8014ab4 nodeName:}" failed. No retries permitted until 2026-05-28 00:02:48.991000293 +0000 UTC m=+1318.380504091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-htx8v" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:02:48.491230 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.491050    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.491478 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.491461    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.502008 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.501980    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8gj\" (UniqueName: \"kubernetes.io/projected/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-kube-api-access-hs8gj\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.993388 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.993353    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.993583 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.993415    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.993583 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.993504    2572 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:02:48.993583 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:02:48.993565    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls podName:4ace6bee-91c4-4a59-a7e1-feb8b8014ab4 nodeName:}" failed. No retries permitted until 2026-05-28 00:02:49.993545715 +0000 UTC m=+1319.383049529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-htx8v" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:02:48.993721 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.993584    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.996037 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.996015    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:48.996159 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:48.996136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:50.000753 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:50.000712    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:50.003262 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:50.003240    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-htx8v\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:50.190256 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:50.190221    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:02:50.323460 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:50.323436    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"]
May 28 00:02:50.324999 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:02:50.324979    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ace6bee_91c4_4a59_a7e1_feb8b8014ab4.slice/crio-3d58158851a46344e4af06b7d3c73a335eb7fa97aa5271867dae4f7c98059877 WatchSource:0}: Error finding container 3d58158851a46344e4af06b7d3c73a335eb7fa97aa5271867dae4f7c98059877: Status 404 returned error can't find the container with id 3d58158851a46344e4af06b7d3c73a335eb7fa97aa5271867dae4f7c98059877
May 28 00:02:51.150107 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:51.150071    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerStarted","Data":"2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0"}
May 28 00:02:51.150107 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:51.150111    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerStarted","Data":"3d58158851a46344e4af06b7d3c73a335eb7fa97aa5271867dae4f7c98059877"}
May 28 00:02:55.164456 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:55.164424    2572 generic.go:358] "Generic (PLEG): container finished" podID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerID="2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0" exitCode=0
May 28 00:02:55.164816 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:55.164472    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerDied","Data":"2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0"}
May 28 00:02:56.168799 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:02:56.168758    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerStarted","Data":"0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9"}
May 28 00:03:01.187807 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:01.187771    2572 generic.go:358] "Generic (PLEG): container finished" podID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerID="0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9" exitCode=0
May 28 00:03:01.188212 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:01.187833    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerDied","Data":"0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9"}
May 28 00:03:02.193907 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:02.193874    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerStarted","Data":"39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3"}
May 28 00:03:02.193907 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:02.193910    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerStarted","Data":"9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114"}
May 28 00:03:02.193907 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:02.193919    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerStarted","Data":"4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3"}
May 28 00:03:02.217960 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:02.217906    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podStartSLOduration=14.217890449 podStartE2EDuration="14.217890449s" podCreationTimestamp="2026-05-28 00:02:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:03:02.21664311 +0000 UTC m=+1331.606146925" watchObservedRunningTime="2026-05-28 00:03:02.217890449 +0000 UTC m=+1331.607394263"
May 28 00:03:05.190881 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:05.190829    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:05.191361 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:05.190893    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:05.191361 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:05.190909    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:05.192621 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:05.192588    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused"
May 28 00:03:05.192769 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:05.192632    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 00:03:05.192769 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:05.192642    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 00:03:08.191513 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.191481    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.191950 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.191667    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.192171 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.192147    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.192258 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.192176    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.192478 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.192456    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.192701 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.192681    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.195945 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.195924    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.213997 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.213931    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:03:08.217006 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:03:08.216982    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:04:09.642268 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.642231    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"]
May 28 00:04:09.642861 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.642571    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online" containerID="cri-o://4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3" gracePeriod=30
May 28 00:04:09.642861 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.642618    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui" containerID="cri-o://39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3" gracePeriod=30
May 28 00:04:09.642861 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.642649    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="offline" containerID="cri-o://9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114" gracePeriod=30
May 28 00:04:09.867360 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.867328    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"]
May 28 00:04:09.867847 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.867686    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="registry" containerID="cri-o://88a05bfde853d62ddde49751f7ffb561f5874047a2fedcc2e77b00a6a7441064" gracePeriod=30
May 28 00:04:09.867847 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.867731    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui" containerID="cri-o://5368ffb34d25b78d1557798979f1d9fc6b030dec3d7befc7176aa7fb40d3a6ae" gracePeriod=30
May 28 00:04:09.867847 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.867731    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="offline" containerID="cri-o://ed56eb0a3a71b951f45ddb3ee8c8d8558ed228702f104965a285a1b71d7d46c6" gracePeriod=30
May 28 00:04:09.867847 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:09.867747    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online" containerID="cri-o://4f9c1daa7122cf997bfbc2814a9a5444b8f38750f8c95ccfcd1f596a0fc9e8c1" gracePeriod=30
May 28 00:04:10.390534 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:10.390507    2572 generic.go:358] "Generic (PLEG): container finished" podID="261de952-7269-41a1-b2c9-d2e931bd8489" containerID="5368ffb34d25b78d1557798979f1d9fc6b030dec3d7befc7176aa7fb40d3a6ae" exitCode=0
May 28 00:04:10.390670 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:10.390579    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"5368ffb34d25b78d1557798979f1d9fc6b030dec3d7befc7176aa7fb40d3a6ae"}
May 28 00:04:10.392880 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:10.392858    2572 generic.go:358] "Generic (PLEG): container finished" podID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerID="39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3" exitCode=0
May 28 00:04:10.392880 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:10.392879    2572 generic.go:358] "Generic (PLEG): container finished" podID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerID="4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3" exitCode=0
May 28 00:04:10.393036 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:10.392903    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerDied","Data":"39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3"}
May 28 00:04:10.393036 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:10.392929    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerDied","Data":"4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3"}
May 28 00:04:11.398423 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:11.398390    2572 generic.go:358] "Generic (PLEG): container finished" podID="261de952-7269-41a1-b2c9-d2e931bd8489" containerID="4f9c1daa7122cf997bfbc2814a9a5444b8f38750f8c95ccfcd1f596a0fc9e8c1" exitCode=0
May 28 00:04:11.398787 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:11.398450    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"4f9c1daa7122cf997bfbc2814a9a5444b8f38750f8c95ccfcd1f596a0fc9e8c1"}
May 28 00:04:15.764407 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:15.764350    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 00:04:15.962192 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:15.962152    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 00:04:18.193075 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:18.193021    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 00:04:18.214212 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:18.214183    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 00:04:25.763977 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:25.763924    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 00:04:25.962803 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:25.962765    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 00:04:28.192657 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:28.192609    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 00:04:28.214828 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:28.214799    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 00:04:35.764328 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:35.764194    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 00:04:35.764764 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:35.764335    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:04:35.962634 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:35.962593    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 00:04:35.962796 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:35.962715    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:04:38.193071 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:38.193024    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 00:04:38.193515 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:38.193150    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:04:38.214417 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:38.214377    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 00:04:38.214537 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:38.214517    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:04:40.291485 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.291464    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:04:40.395525 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.395496    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") pod \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") "
May 28 00:04:40.395525 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.395529    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls\") pod \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") "
May 28 00:04:40.395902 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.395563    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs8gj\" (UniqueName: \"kubernetes.io/projected/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-kube-api-access-hs8gj\") pod \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") "
May 28 00:04:40.395902 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.395638    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-registry-tls\") pod \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") "
May 28 00:04:40.395902 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.395656    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-feast-data\") pod \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") "
May 28 00:04:40.395902 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.395707    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls\") pod \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\" (UID: \"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4\") "
May 28 00:04:40.396119 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.396060    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:04:40.396520 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.396466    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-feast-data" (OuterVolumeSpecName: "feast-data") pod "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:04:40.398631 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.398603    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.398631 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.398619    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls" (OuterVolumeSpecName: "online-tls") pod "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.398806 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.398621    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-kube-api-access-hs8gj" (OuterVolumeSpecName: "kube-api-access-hs8gj") pod "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4"). InnerVolumeSpecName "kube-api-access-hs8gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:04:40.398806 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.398658    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" (UID: "4ace6bee-91c4-4a59-a7e1-feb8b8014ab4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.478608 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.478570    2572 generic.go:358] "Generic (PLEG): container finished" podID="261de952-7269-41a1-b2c9-d2e931bd8489" containerID="ed56eb0a3a71b951f45ddb3ee8c8d8558ed228702f104965a285a1b71d7d46c6" exitCode=137
May 28 00:04:40.478608 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.478605    2572 generic.go:358] "Generic (PLEG): container finished" podID="261de952-7269-41a1-b2c9-d2e931bd8489" containerID="88a05bfde853d62ddde49751f7ffb561f5874047a2fedcc2e77b00a6a7441064" exitCode=137
May 28 00:04:40.478822 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.478633    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"ed56eb0a3a71b951f45ddb3ee8c8d8558ed228702f104965a285a1b71d7d46c6"}
May 28 00:04:40.478822 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.478666    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"88a05bfde853d62ddde49751f7ffb561f5874047a2fedcc2e77b00a6a7441064"}
May 28 00:04:40.481232 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.481208    2572 generic.go:358] "Generic (PLEG): container finished" podID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerID="9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114" exitCode=137
May 28 00:04:40.481367 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.481269    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerDied","Data":"9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114"}
May 28 00:04:40.481367 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.481295    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v" event={"ID":"4ace6bee-91c4-4a59-a7e1-feb8b8014ab4","Type":"ContainerDied","Data":"3d58158851a46344e4af06b7d3c73a335eb7fa97aa5271867dae4f7c98059877"}
May 28 00:04:40.481367 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.481323    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"
May 28 00:04:40.481367 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.481335    2572 scope.go:117] "RemoveContainer" containerID="39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3"
May 28 00:04:40.490391 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.490335    2572 scope.go:117] "RemoveContainer" containerID="9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114"
May 28 00:04:40.496629 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.496608    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.496629 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.496629    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.496805 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.496639    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.496805 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.496647    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.496805 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.496656    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.496805 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.496665    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8gj\" (UniqueName: \"kubernetes.io/projected/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4-kube-api-access-hs8gj\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.499797 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.499778    2572 scope.go:117] "RemoveContainer" containerID="4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3"
May 28 00:04:40.504823 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.504798    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"]
May 28 00:04:40.507700 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.507681    2572 scope.go:117] "RemoveContainer" containerID="0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9"
May 28 00:04:40.511034 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.511013    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-htx8v"]
May 28 00:04:40.516063 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.515988    2572 scope.go:117] "RemoveContainer" containerID="2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0"
May 28 00:04:40.516506 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.516491    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:04:40.522590 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.522570    2572 scope.go:117] "RemoveContainer" containerID="39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3"
May 28 00:04:40.522833 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:04:40.522814    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3\": container with ID starting with 39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3 not found: ID does not exist" containerID="39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3"
May 28 00:04:40.522876 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.522848    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3"} err="failed to get container status \"39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3\": rpc error: code = NotFound desc = could not find container \"39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3\": container with ID starting with 39098d2148d90568d24e157d706ff2c8598b6054ab047288aab16774a3d91aa3 not found: ID does not exist"
May 28 00:04:40.522876 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.522873    2572 scope.go:117] "RemoveContainer" containerID="9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114"
May 28 00:04:40.523106 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:04:40.523091    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114\": container with ID starting with 9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114 not found: ID does not exist" containerID="9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114"
May 28 00:04:40.523160 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523112    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114"} err="failed to get container status \"9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114\": rpc error: code = NotFound desc = could not find container \"9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114\": container with ID starting with 9b96e9586093e5e8b95587fe540c7040fff3aba4b984302096a7d7ac21884114 not found: ID does not exist"
May 28 00:04:40.523160 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523127    2572 scope.go:117] "RemoveContainer" containerID="4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3"
May 28 00:04:40.523372 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:04:40.523354    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3\": container with ID starting with 4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3 not found: ID does not exist" containerID="4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3"
May 28 00:04:40.523426 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523381    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3"} err="failed to get container status \"4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3\": rpc error: code = NotFound desc = could not find container \"4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3\": container with ID starting with 4c16a2bf9228c8d020e08c7ac125a9369508302989be2c99f55e6a9e5356e9c3 not found: ID does not exist"
May 28 00:04:40.523426 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523397    2572 scope.go:117] "RemoveContainer" containerID="0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9"
May 28 00:04:40.523588 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:04:40.523574    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9\": container with ID starting with 0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9 not found: ID does not exist" containerID="0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9"
May 28 00:04:40.523624 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523593    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9"} err="failed to get container status \"0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9\": rpc error: code = NotFound desc = could not find container \"0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9\": container with ID starting with 0a3c31f8f9f11df118446b53b5fc42b5116877cff51a128c783758631eb761c9 not found: ID does not exist"
May 28 00:04:40.523624 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523606    2572 scope.go:117] "RemoveContainer" containerID="2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0"
May 28 00:04:40.523796 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:04:40.523778    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0\": container with ID starting with 2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0 not found: ID does not exist" containerID="2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0"
May 28 00:04:40.523845 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.523800    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0"} err="failed to get container status \"2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0\": rpc error: code = NotFound desc = could not find container \"2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0\": container with ID starting with 2543f02ca424ba18065dba7211cf6baf5627fca605daeeece756aed2fa86bda0 not found: ID does not exist"
May 28 00:04:40.597652 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.597627    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-registry-tls\") pod \"261de952-7269-41a1-b2c9-d2e931bd8489\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") "
May 28 00:04:40.597905 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.597664    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brxtc\" (UniqueName: \"kubernetes.io/projected/261de952-7269-41a1-b2c9-d2e931bd8489-kube-api-access-brxtc\") pod \"261de952-7269-41a1-b2c9-d2e931bd8489\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") "
May 28 00:04:40.597905 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.597699    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/261de952-7269-41a1-b2c9-d2e931bd8489-feast-data\") pod \"261de952-7269-41a1-b2c9-d2e931bd8489\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") "
May 28 00:04:40.597905 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.597721    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-ui-tls\") pod \"261de952-7269-41a1-b2c9-d2e931bd8489\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") "
May 28 00:04:40.597905 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.597834    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls\") pod \"261de952-7269-41a1-b2c9-d2e931bd8489\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") "
May 28 00:04:40.597905 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.597878    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls\") pod \"261de952-7269-41a1-b2c9-d2e931bd8489\" (UID: \"261de952-7269-41a1-b2c9-d2e931bd8489\") "
May 28 00:04:40.598251 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.598226    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261de952-7269-41a1-b2c9-d2e931bd8489-feast-data" (OuterVolumeSpecName: "feast-data") pod "261de952-7269-41a1-b2c9-d2e931bd8489" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:04:40.600078 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.600050    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "261de952-7269-41a1-b2c9-d2e931bd8489" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.600168 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.600073    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "261de952-7269-41a1-b2c9-d2e931bd8489" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.600168 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.600100    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "261de952-7269-41a1-b2c9-d2e931bd8489" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.600492 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.600469    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261de952-7269-41a1-b2c9-d2e931bd8489-kube-api-access-brxtc" (OuterVolumeSpecName: "kube-api-access-brxtc") pod "261de952-7269-41a1-b2c9-d2e931bd8489" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489"). InnerVolumeSpecName "kube-api-access-brxtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:04:40.600628 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.600612    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls" (OuterVolumeSpecName: "online-tls") pod "261de952-7269-41a1-b2c9-d2e931bd8489" (UID: "261de952-7269-41a1-b2c9-d2e931bd8489"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:04:40.699321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.699243    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.699321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.699266    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brxtc\" (UniqueName: \"kubernetes.io/projected/261de952-7269-41a1-b2c9-d2e931bd8489-kube-api-access-brxtc\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.699321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.699276    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/261de952-7269-41a1-b2c9-d2e931bd8489-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.699321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.699287    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.699321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.699295    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:40.699321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:40.699323    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/261de952-7269-41a1-b2c9-d2e931bd8489-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:04:41.220922 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.220894    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" path="/var/lib/kubelet/pods/4ace6bee-91c4-4a59-a7e1-feb8b8014ab4/volumes"
May 28 00:04:41.486378 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.486273    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv" event={"ID":"261de952-7269-41a1-b2c9-d2e931bd8489","Type":"ContainerDied","Data":"fad08cc68200e875fae2642484ae38704a6e1c735d699aa6f6ebff9d820c1153"}
May 28 00:04:41.486378 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.486301    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"
May 28 00:04:41.486378 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.486335    2572 scope.go:117] "RemoveContainer" containerID="5368ffb34d25b78d1557798979f1d9fc6b030dec3d7befc7176aa7fb40d3a6ae"
May 28 00:04:41.494418 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.494402    2572 scope.go:117] "RemoveContainer" containerID="ed56eb0a3a71b951f45ddb3ee8c8d8558ed228702f104965a285a1b71d7d46c6"
May 28 00:04:41.501360 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.501341    2572 scope.go:117] "RemoveContainer" containerID="4f9c1daa7122cf997bfbc2814a9a5444b8f38750f8c95ccfcd1f596a0fc9e8c1"
May 28 00:04:41.512817 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.512794    2572 scope.go:117] "RemoveContainer" containerID="88a05bfde853d62ddde49751f7ffb561f5874047a2fedcc2e77b00a6a7441064"
May 28 00:04:41.515470 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.515441    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"]
May 28 00:04:41.521566 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.521547    2572 scope.go:117] "RemoveContainer" containerID="3993540e687dcb8b1ee3c4f4adf00583dfb141ff973a2a4db40a3fe4fb40c9c0"
May 28 00:04:41.523920 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.523898    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-js9mv"]
May 28 00:04:41.528664 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:41.528648    2572 scope.go:117] "RemoveContainer" containerID="17fac3d3ab40d491cf56b6f659347acea8093acac08b77376138f9a2c6f0ba9e"
May 28 00:04:43.219540 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:04:43.219508    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" path="/var/lib/kubelet/pods/261de952-7269-41a1-b2c9-d2e931bd8489/volumes"
May 28 00:05:19.878392 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878355    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"]
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878634    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="offline"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878646    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="offline"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878656    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="feast-init"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878662    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="feast-init"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878674    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878680    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878688    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="feast-init"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878694    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="feast-init"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878700    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="feast-apply"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878705    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="feast-apply"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878711    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="feast-apply"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878716    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="feast-apply"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878720    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878725    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878732    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878737    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878743    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="offline"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878748    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="offline"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878753    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="registry"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878758    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="registry"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878764    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878768    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878806    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="offline"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878814    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="offline"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878822    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="ui"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878827    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ace6bee-91c4-4a59-a7e1-feb8b8014ab4" containerName="online"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878834    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="ui"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878841    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="registry"
May 28 00:05:19.878834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.878848    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="261de952-7269-41a1-b2c9-d2e931bd8489" containerName="online"
May 28 00:05:19.882041 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.882024    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:19.884863 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.884832    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 00:05:19.884985 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.884879    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 00:05:19.886042 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.886018    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-j7qdh\""
May 28 00:05:19.886200 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.886077    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 00:05:19.886200 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.886122    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 00:05:19.886200 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.886186    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 00:05:19.886429 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.886414    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 00:05:19.899438 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.899414    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"]
May 28 00:05:19.975478 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.975453    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnllm\" (UniqueName: \"kubernetes.io/projected/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-kube-api-access-gnllm\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:19.975615 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.975494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:19.975615 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.975523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:19.975615 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.975591    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:19.975720 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.975634    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:19.975720 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:19.975700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.076585 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.076549    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.076682 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.076595    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gnllm\" (UniqueName: \"kubernetes.io/projected/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-kube-api-access-gnllm\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.076682 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.076615    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.076682 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.076656    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.076682 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.076681    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.076810 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.076709    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.077158 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.077122    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.079176 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.079148    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.079287 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.079231    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.079385 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.079292    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.079385 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.079369    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.085688 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.085663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnllm\" (UniqueName: \"kubernetes.io/projected/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-kube-api-access-gnllm\") pod \"feast-simple-feast-setup-664fcfcc58-7kgrj\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.201704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.201667    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:20.332089 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.332063    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"]
May 28 00:05:20.334577 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:05:20.334537    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod065ec6d7_89ce_43a1_bd6a_1e7a39e9b30b.slice/crio-4c460f3bb1e8e2e901564fb863b6419802def2950619f3aa294f8193e98ec30d WatchSource:0}: Error finding container 4c460f3bb1e8e2e901564fb863b6419802def2950619f3aa294f8193e98ec30d: Status 404 returned error can't find the container with id 4c460f3bb1e8e2e901564fb863b6419802def2950619f3aa294f8193e98ec30d
May 28 00:05:20.336384 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.336365    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:05:20.605424 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.605327    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"}
May 28 00:05:20.605424 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:20.605379    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"4c460f3bb1e8e2e901564fb863b6419802def2950619f3aa294f8193e98ec30d"}
May 28 00:05:25.620289 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:25.620251    2572 generic.go:358] "Generic (PLEG): container finished" podID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerID="c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a" exitCode=0
May 28 00:05:25.620913 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:25.620337    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"}
May 28 00:05:26.624888 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:26.624851    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"}
May 28 00:05:31.640382 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:31.640341    2572 generic.go:358] "Generic (PLEG): container finished" podID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerID="a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f" exitCode=0
May 28 00:05:31.640771 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:31.640395    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"}
May 28 00:05:32.647941 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:32.647894    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"}
May 28 00:05:32.648452 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:32.647953    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"}
May 28 00:05:32.648452 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:32.647971    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"}
May 28 00:05:32.648452 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:32.647986    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerStarted","Data":"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"}
May 28 00:05:32.682399 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:32.682326    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podStartSLOduration=13.682287819999999 podStartE2EDuration="13.68228782s" podCreationTimestamp="2026-05-28 00:05:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:05:32.680399995 +0000 UTC m=+1482.069903810" watchObservedRunningTime="2026-05-28 00:05:32.68228782 +0000 UTC m=+1482.071791634"
May 28 00:05:35.202128 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.202090    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:35.202719 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.202697    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:35.202854 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.202836    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:35.202995 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.202885    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:35.204178 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.204146    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused"
May 28 00:05:35.204339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.204173    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused"
May 28 00:05:35.204437 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.204207    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 28 00:05:35.204508 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:35.204254    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 28 00:05:38.203658 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.203620    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204112 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.203902    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204112 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204014    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204349 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204324    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204479 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204360    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204479 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204375    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204479 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204461    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204713 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204688    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.204820 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.204781    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.208156 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.208140    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.667475 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.667450    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:38.670364 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:38.670343    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:05:51.209580 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:51.209543    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:05:51.211108 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:05:51.211084    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:06:41.177502 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.177465    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"]
May 28 00:06:41.178003 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.177925    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="registry" containerID="cri-o://0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027" gracePeriod=30
May 28 00:06:41.178086 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.177973    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui" containerID="cri-o://4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c" gracePeriod=30
May 28 00:06:41.178086 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.177998    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="offline" containerID="cri-o://05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e" gracePeriod=30
May 28 00:06:41.178086 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.178062    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online" containerID="cri-o://fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4" gracePeriod=30
May 28 00:06:41.570008 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.569921    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"]
May 28 00:06:41.573671 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.573651    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.576133 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.576105    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-m75fx\""
May 28 00:06:41.586872 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.586847    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"]
May 28 00:06:41.736725 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.736651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.736725 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.736723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92m7t\" (UniqueName: \"kubernetes.io/projected/98c0a07e-96cf-4aa0-a610-322683d457f0-kube-api-access-92m7t\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.736997 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.736751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.736997 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.736773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.736997 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.736853    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/98c0a07e-96cf-4aa0-a610-322683d457f0-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.736997 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.736919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838351 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838238    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838351 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838324    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838351 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838348    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-92m7t\" (UniqueName: \"kubernetes.io/projected/98c0a07e-96cf-4aa0-a610-322683d457f0-kube-api-access-92m7t\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838641 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838373    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838641 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838401    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838641 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838439    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/98c0a07e-96cf-4aa0-a610-322683d457f0-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.838838 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.838812    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/98c0a07e-96cf-4aa0-a610-322683d457f0-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.841202 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.841171    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.841359 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.841221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.841431 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.841395    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.841520 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.841499    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.846206 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.846179    2572 generic.go:358] "Generic (PLEG): container finished" podID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerID="4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c" exitCode=0
May 28 00:06:41.846355 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.846242    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"}
May 28 00:06:41.847041 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.847017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92m7t\" (UniqueName: \"kubernetes.io/projected/98c0a07e-96cf-4aa0-a610-322683d457f0-kube-api-access-92m7t\") pod \"feast-simple-feast-setup-664fcfcc58-x6r67\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:41.882912 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:41.882875    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:42.012611 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:42.012583    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"]
May 28 00:06:42.015041 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:06:42.015013    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98c0a07e_96cf_4aa0_a610_322683d457f0.slice/crio-0cebd2f64d4925846870450cc269c84b07b3342d4206888d61cae04c31aa16fa WatchSource:0}: Error finding container 0cebd2f64d4925846870450cc269c84b07b3342d4206888d61cae04c31aa16fa: Status 404 returned error can't find the container with id 0cebd2f64d4925846870450cc269c84b07b3342d4206888d61cae04c31aa16fa
May 28 00:06:42.852662 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:42.852631    2572 generic.go:358] "Generic (PLEG): container finished" podID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerID="fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4" exitCode=0
May 28 00:06:42.853053 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:42.852714    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"}
May 28 00:06:42.854081 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:42.854057    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"0c9abce127c8d8a6e824260f63ed844ef5dfad6f0248e735a2dfe6229b05d7e0"}
May 28 00:06:42.854180 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:42.854088    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"0cebd2f64d4925846870450cc269c84b07b3342d4206888d61cae04c31aa16fa"}
May 28 00:06:46.868170 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:46.868077    2572 generic.go:358] "Generic (PLEG): container finished" podID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerID="0c9abce127c8d8a6e824260f63ed844ef5dfad6f0248e735a2dfe6229b05d7e0" exitCode=0
May 28 00:06:46.868170 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:46.868133    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"0c9abce127c8d8a6e824260f63ed844ef5dfad6f0248e735a2dfe6229b05d7e0"}
May 28 00:06:47.873218 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:47.873181    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"27efc2e69f67b6e5b707b3d6e7b33176ea2c816476296fe478af2adab158a35f"}
May 28 00:06:48.205068 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:48.205020    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 28 00:06:48.667738 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:48.667704    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 28 00:06:52.889582 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:52.889546    2572 generic.go:358] "Generic (PLEG): container finished" podID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerID="27efc2e69f67b6e5b707b3d6e7b33176ea2c816476296fe478af2adab158a35f" exitCode=0
May 28 00:06:52.889946 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:52.889590    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"27efc2e69f67b6e5b707b3d6e7b33176ea2c816476296fe478af2adab158a35f"}
May 28 00:06:53.896909 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:53.896862    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"4c065f7b4bbedaf889c3cd94a0aaa06f79b6e1bdd95892f83308c46850009e07"}
May 28 00:06:53.897409 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:53.896926    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"a1d0d4fa77fbd1c9edf82acd04db499ab5485e6f2510c8d2e11a602eeecdbd2a"}
May 28 00:06:53.897409 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:53.896945    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"5e59f6b95c0a8c7b21f78903f4f16da979149acb1923de3755a034e425fbcefd"}
May 28 00:06:53.897409 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:53.896958    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerStarted","Data":"5e7d8c4000faeec473f5fc41bcf96bb0e9a27eae87aff303265d51d0ad732842"}
May 28 00:06:53.921953 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:53.921893    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podStartSLOduration=12.921873236 podStartE2EDuration="12.921873236s" podCreationTimestamp="2026-05-28 00:06:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:06:53.920536952 +0000 UTC m=+1563.310040798" watchObservedRunningTime="2026-05-28 00:06:53.921873236 +0000 UTC m=+1563.311377049"
May 28 00:06:56.883209 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.883168    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:56.883703 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.883227    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:56.883703 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.883240    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:56.883703 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.883250    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:56.885425 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.885390    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused"
May 28 00:06:56.885657 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.885438    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 00:06:56.885777 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.885460    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused"
May 28 00:06:56.885777 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:56.885494    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused"
May 28 00:06:58.205184 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:58.205139    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 28 00:06:58.668057 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:58.668018    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 28 00:06:59.884500 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.884298    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.884953 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.884579    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.884953 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.884693    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.885070 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.885011    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.885070 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.885046    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.885070 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.885061    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.885273 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.885253    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.885392 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.885376    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.885455 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.885420    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.889082 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.889062    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.916820 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.916799    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:06:59.919395 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:06:59.919378    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:07:08.205269 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:08.205226    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 28 00:07:08.205761 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:08.205422    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:07:08.668082 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:08.668020    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 28 00:07:08.668229 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:08.668149    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:07:11.822257 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.822233    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:07:11.875810 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.875782    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-ui-tls\") pod \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") "
May 28 00:07:11.875973 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.875821    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnllm\" (UniqueName: \"kubernetes.io/projected/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-kube-api-access-gnllm\") pod \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") "
May 28 00:07:11.875973 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.875850    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-registry-tls\") pod \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") "
May 28 00:07:11.875973 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.875872    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-offline-tls\") pod \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") "
May 28 00:07:11.875973 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.875946    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-online-tls\") pod \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") "
May 28 00:07:11.876202 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.875983    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-feast-data\") pod \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\" (UID: \"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b\") "
May 28 00:07:11.876834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.876795    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-feast-data" (OuterVolumeSpecName: "feast-data") pod "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" (UID: "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:07:11.878273 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.878244    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-online-tls" (OuterVolumeSpecName: "online-tls") pod "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" (UID: "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:07:11.878381 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.878263    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" (UID: "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:07:11.878381 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.878354    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" (UID: "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:07:11.878546 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.878526    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-kube-api-access-gnllm" (OuterVolumeSpecName: "kube-api-access-gnllm") pod "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" (UID: "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b"). InnerVolumeSpecName "kube-api-access-gnllm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:07:11.878614 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.878601    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" (UID: "065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:07:11.951650 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951577    2572 generic.go:358] "Generic (PLEG): container finished" podID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerID="05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e" exitCode=137
May 28 00:07:11.951650 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951607    2572 generic.go:358] "Generic (PLEG): container finished" podID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerID="0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027" exitCode=137
May 28 00:07:11.951650 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951614    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"}
May 28 00:07:11.951828 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951657    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"}
May 28 00:07:11.951828 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951661    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"
May 28 00:07:11.951828 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951668    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj" event={"ID":"065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b","Type":"ContainerDied","Data":"4c460f3bb1e8e2e901564fb863b6419802def2950619f3aa294f8193e98ec30d"}
May 28 00:07:11.951828 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.951684    2572 scope.go:117] "RemoveContainer" containerID="4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"
May 28 00:07:11.960055 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.960039    2572 scope.go:117] "RemoveContainer" containerID="05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"
May 28 00:07:11.967757 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.967740    2572 scope.go:117] "RemoveContainer" containerID="fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"
May 28 00:07:11.973643 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.973620    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"]
May 28 00:07:11.974918 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.974899    2572 scope.go:117] "RemoveContainer" containerID="0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"
May 28 00:07:11.977462 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977437    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:07:11.977567 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977469    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnllm\" (UniqueName: \"kubernetes.io/projected/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-kube-api-access-gnllm\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:07:11.977567 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977486    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:07:11.977567 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977501    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:07:11.977567 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977514    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:07:11.977567 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977527    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:07:11.977927 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.977722    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7kgrj"]
May 28 00:07:11.982346 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.982323    2572 scope.go:117] "RemoveContainer" containerID="a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"
May 28 00:07:11.990494 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.990478    2572 scope.go:117] "RemoveContainer" containerID="c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"
May 28 00:07:11.997024 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997007    2572 scope.go:117] "RemoveContainer" containerID="4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"
May 28 00:07:11.997252 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:07:11.997235    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c\": container with ID starting with 4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c not found: ID does not exist" containerID="4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"
May 28 00:07:11.997339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997259    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"} err="failed to get container status \"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c\": rpc error: code = NotFound desc = could not find container \"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c\": container with ID starting with 4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c not found: ID does not exist"
May 28 00:07:11.997339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997276    2572 scope.go:117] "RemoveContainer" containerID="05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"
May 28 00:07:11.997547 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:07:11.997529    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e\": container with ID starting with 05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e not found: ID does not exist" containerID="05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"
May 28 00:07:11.997591 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997555    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"} err="failed to get container status \"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e\": rpc error: code = NotFound desc = could not find container \"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e\": container with ID starting with 05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e not found: ID does not exist"
May 28 00:07:11.997591 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997573    2572 scope.go:117] "RemoveContainer" containerID="fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"
May 28 00:07:11.997811 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:07:11.997794    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4\": container with ID starting with fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4 not found: ID does not exist" containerID="fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"
May 28 00:07:11.997884 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997820    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"} err="failed to get container status \"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4\": rpc error: code = NotFound desc = could not find container \"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4\": container with ID starting with fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4 not found: ID does not exist"
May 28 00:07:11.997884 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.997843    2572 scope.go:117] "RemoveContainer" containerID="0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"
May 28 00:07:11.998090 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:07:11.998073    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027\": container with ID starting with 0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027 not found: ID does not exist" containerID="0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"
May 28 00:07:11.998125 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998096    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"} err="failed to get container status \"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027\": rpc error: code = NotFound desc = could not find container \"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027\": container with ID starting with 0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027 not found: ID does not exist"
May 28 00:07:11.998125 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998110    2572 scope.go:117] "RemoveContainer" containerID="a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"
May 28 00:07:11.998328 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:07:11.998295    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f\": container with ID starting with a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f not found: ID does not exist" containerID="a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"
May 28 00:07:11.998371 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998334    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"} err="failed to get container status \"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f\": rpc error: code = NotFound desc = could not find container \"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f\": container with ID starting with a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f not found: ID does not exist"
May 28 00:07:11.998371 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998350    2572 scope.go:117] "RemoveContainer" containerID="c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"
May 28 00:07:11.998571 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:07:11.998557    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a\": container with ID starting with c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a not found: ID does not exist" containerID="c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"
May 28 00:07:11.998612 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998575    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"} err="failed to get container status \"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a\": rpc error: code = NotFound desc = could not find container \"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a\": container with ID starting with c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a not found: ID does not exist"
May 28 00:07:11.998612 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998588    2572 scope.go:117] "RemoveContainer" containerID="4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"
May 28 00:07:11.998825 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998808    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c"} err="failed to get container status \"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c\": rpc error: code = NotFound desc = could not find container \"4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c\": container with ID starting with 4423ff5a46cfcb996200a47b8c57c863d037799726e263ea63d66f65a1c44e8c not found: ID does not exist"
May 28 00:07:11.998868 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.998827    2572 scope.go:117] "RemoveContainer" containerID="05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"
May 28 00:07:11.999057 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999038    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e"} err="failed to get container status \"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e\": rpc error: code = NotFound desc = could not find container \"05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e\": container with ID starting with 05a1ec4a51278228cb426ce7bdaf67ee7b1d84198874d0b8c061f31b0b1c3f1e not found: ID does not exist"
May 28 00:07:11.999101 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999058    2572 scope.go:117] "RemoveContainer" containerID="fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"
May 28 00:07:11.999256 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999236    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4"} err="failed to get container status \"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4\": rpc error: code = NotFound desc = could not find container \"fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4\": container with ID starting with fdf11f0ffeb2691e528198e3a857f29e96556d55d3c24d45c84c30f10ec60aa4 not found: ID does not exist"
May 28 00:07:11.999342 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999258    2572 scope.go:117] "RemoveContainer" containerID="0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"
May 28 00:07:11.999467 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999452    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027"} err="failed to get container status \"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027\": rpc error: code = NotFound desc = could not find container \"0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027\": container with ID starting with 0c2516692143d82691f420467a393805c2a3d5d147dd7737fa0e8021e20fa027 not found: ID does not exist"
May 28 00:07:11.999509 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999469    2572 scope.go:117] "RemoveContainer" containerID="a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"
May 28 00:07:11.999690 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999673    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f"} err="failed to get container status \"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f\": rpc error: code = NotFound desc = could not find container \"a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f\": container with ID starting with a1ab9c4660f893fdf3676f65917db427502cd85a525599ff7079aec96078420f not found: ID does not exist"
May 28 00:07:11.999733 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999692    2572 scope.go:117] "RemoveContainer" containerID="c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"
May 28 00:07:11.999897 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:11.999879    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a"} err="failed to get container status \"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a\": rpc error: code = NotFound desc = could not find container \"c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a\": container with ID starting with c631169b10d4f5dcec02c75128d3d0e758df561a4294a2a094442b624ae4ad7a not found: ID does not exist"
May 28 00:07:13.219397 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:07:13.219363    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" path="/var/lib/kubelet/pods/065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b/volumes"
May 28 00:08:02.872944 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.872897    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"]
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873183    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873193    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873204    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="offline"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873210    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="offline"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873225    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873230    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873238    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="feast-apply"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873243    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="feast-apply"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873251    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="feast-init"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873255    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="feast-init"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873262    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="registry"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873267    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="registry"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873333    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="offline"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873346    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="ui"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873352    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="registry"
May 28 00:08:02.873402 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.873359    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="065ec6d7-89ce-43a1-bd6a-1e7a39e9b30b" containerName="online"
May 28 00:08:02.877920 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.877903    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:02.880504 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.880477    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 00:08:02.880656 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.880511    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 00:08:02.880656 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.880522    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 00:08:02.880656 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.880556    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 00:08:02.880656 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.880637    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-d9xm6\""
May 28 00:08:02.880858 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.880704    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 00:08:02.881175 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.881161    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 00:08:02.889734 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.889679    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"]
May 28 00:08:02.971705 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.971663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/687a3b3a-eb87-4844-987a-b941fec5edf4-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:02.971926 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.971730    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:02.971926 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.971768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:02.971926 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.971796    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wh5\" (UniqueName: \"kubernetes.io/projected/687a3b3a-eb87-4844-987a-b941fec5edf4-kube-api-access-l2wh5\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:02.971926 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.971850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/687a3b3a-eb87-4844-987a-b941fec5edf4-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:02.972112 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:02.971966    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.072820 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.072783    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/687a3b3a-eb87-4844-987a-b941fec5edf4-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.072993 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.072828    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.072993 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.072953    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.073115 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.073001    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wh5\" (UniqueName: \"kubernetes.io/projected/687a3b3a-eb87-4844-987a-b941fec5edf4-kube-api-access-l2wh5\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.073115 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.073035    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/687a3b3a-eb87-4844-987a-b941fec5edf4-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.073115 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.073069    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.073289 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.073227    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/687a3b3a-eb87-4844-987a-b941fec5edf4-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.073740 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.073712    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/687a3b3a-eb87-4844-987a-b941fec5edf4-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.075565 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.075542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.075662 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.075541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.075662 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.075589    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.081105 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.081086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wh5\" (UniqueName: \"kubernetes.io/projected/687a3b3a-eb87-4844-987a-b941fec5edf4-kube-api-access-l2wh5\") pod \"feast-simple-feast-remote-setup-5cff94746b-pm8wf\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.187445 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.187416    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:03.320943 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:03.320918    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"]
May 28 00:08:03.323193 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:08:03.323156    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod687a3b3a_eb87_4844_987a_b941fec5edf4.slice/crio-d92e0e0a189201c29719c65e0cb7b36ec728add1ddd03931c0e39c6d6f35e705 WatchSource:0}: Error finding container d92e0e0a189201c29719c65e0cb7b36ec728add1ddd03931c0e39c6d6f35e705: Status 404 returned error can't find the container with id d92e0e0a189201c29719c65e0cb7b36ec728add1ddd03931c0e39c6d6f35e705
May 28 00:08:04.095820 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:04.095775    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerStarted","Data":"ded74cb79cc342aee9acc90eb7ebec78bdb9640aa39c98e3120d726c118b20af"}
May 28 00:08:04.095820 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:04.095818    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerStarted","Data":"d92e0e0a189201c29719c65e0cb7b36ec728add1ddd03931c0e39c6d6f35e705"}
May 28 00:08:08.109055 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:08.109022    2572 generic.go:358] "Generic (PLEG): container finished" podID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerID="ded74cb79cc342aee9acc90eb7ebec78bdb9640aa39c98e3120d726c118b20af" exitCode=0
May 28 00:08:08.109543 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:08.109099    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerDied","Data":"ded74cb79cc342aee9acc90eb7ebec78bdb9640aa39c98e3120d726c118b20af"}
May 28 00:08:09.112971 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:09.112936    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerStarted","Data":"f7fbea439660103b03ed9d56d80de2ac22cfa843902da0693f1cbd22fcadd0f6"}
May 28 00:08:14.129562 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:14.129525    2572 generic.go:358] "Generic (PLEG): container finished" podID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerID="f7fbea439660103b03ed9d56d80de2ac22cfa843902da0693f1cbd22fcadd0f6" exitCode=0
May 28 00:08:14.129923 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:14.129572    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerDied","Data":"f7fbea439660103b03ed9d56d80de2ac22cfa843902da0693f1cbd22fcadd0f6"}
May 28 00:08:15.137552 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.137511    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerStarted","Data":"ea583a415d3e7f3995f47b8f955b4b5ad2b82f5ccf646bce80e311f11ffecbc8"}
May 28 00:08:15.138019 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.137568    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerStarted","Data":"97e0e306296c7309e6bace76f750f1b98109d53a6a8bdb81e3502c6b262a0c8c"}
May 28 00:08:15.138019 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.137584    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerStarted","Data":"5d7544f6f37007509b64d1de6b340bc65eab2cab91d7c786359ce4bdc45b3dad"}
May 28 00:08:15.161350 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.161271    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podStartSLOduration=13.161246938 podStartE2EDuration="13.161246938s" podCreationTimestamp="2026-05-28 00:08:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:08:15.159710578 +0000 UTC m=+1644.549214420" watchObservedRunningTime="2026-05-28 00:08:15.161246938 +0000 UTC m=+1644.550750753"
May 28 00:08:15.188559 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.188505    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:15.188559 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.188552    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:15.188559 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.188565    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:15.189793 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.189754    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 28 00:08:15.189793 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.189776    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused"
May 28 00:08:15.189955 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:15.189815    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 00:08:18.188257 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:18.188204    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 28 00:08:18.188257 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:18.188233    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 00:08:18.188826 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:18.188231    2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused"
May 28 00:08:21.189535 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.189498    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:21.189926 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.189613    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:21.190036 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.190016    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:21.190172 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.190042    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:21.190251 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.190235    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:21.190417 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.190399    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:21.193497 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:21.193474    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:22.159689 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:22.159650    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:08:22.162595 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:08:22.162575    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:09:23.840410 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:23.840364    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"]
May 28 00:09:23.840889 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:23.840736    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online" containerID="cri-o://5d7544f6f37007509b64d1de6b340bc65eab2cab91d7c786359ce4bdc45b3dad" gracePeriod=30
May 28 00:09:23.840889 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:23.840774    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui" containerID="cri-o://ea583a415d3e7f3995f47b8f955b4b5ad2b82f5ccf646bce80e311f11ffecbc8" gracePeriod=30
May 28 00:09:23.840889 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:23.840809    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="offline" containerID="cri-o://97e0e306296c7309e6bace76f750f1b98109d53a6a8bdb81e3502c6b262a0c8c" gracePeriod=30
May 28 00:09:24.030870 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:24.030835    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"]
May 28 00:09:24.031394 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:24.031349    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="offline" containerID="cri-o://a1d0d4fa77fbd1c9edf82acd04db499ab5485e6f2510c8d2e11a602eeecdbd2a" gracePeriod=30
May 28 00:09:24.031394 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:24.031365    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online" containerID="cri-o://5e59f6b95c0a8c7b21f78903f4f16da979149acb1923de3755a034e425fbcefd" gracePeriod=30
May 28 00:09:24.031592 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:24.031332    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui" containerID="cri-o://4c065f7b4bbedaf889c3cd94a0aaa06f79b6e1bdd95892f83308c46850009e07" gracePeriod=30
May 28 00:09:24.031592 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:24.031508    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="registry" containerID="cri-o://5e7d8c4000faeec473f5fc41bcf96bb0e9a27eae87aff303265d51d0ad732842" gracePeriod=30
May 28 00:09:25.340624 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.340589    2572 generic.go:358] "Generic (PLEG): container finished" podID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerID="ea583a415d3e7f3995f47b8f955b4b5ad2b82f5ccf646bce80e311f11ffecbc8" exitCode=0
May 28 00:09:25.340624 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.340616    2572 generic.go:358] "Generic (PLEG): container finished" podID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerID="5d7544f6f37007509b64d1de6b340bc65eab2cab91d7c786359ce4bdc45b3dad" exitCode=0
May 28 00:09:25.341065 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.340662    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerDied","Data":"ea583a415d3e7f3995f47b8f955b4b5ad2b82f5ccf646bce80e311f11ffecbc8"}
May 28 00:09:25.341065 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.340702    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerDied","Data":"5d7544f6f37007509b64d1de6b340bc65eab2cab91d7c786359ce4bdc45b3dad"}
May 28 00:09:25.343229 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.343206    2572 generic.go:358] "Generic (PLEG): container finished" podID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerID="4c065f7b4bbedaf889c3cd94a0aaa06f79b6e1bdd95892f83308c46850009e07" exitCode=0
May 28 00:09:25.343229 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.343225    2572 generic.go:358] "Generic (PLEG): container finished" podID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerID="5e59f6b95c0a8c7b21f78903f4f16da979149acb1923de3755a034e425fbcefd" exitCode=0
May 28 00:09:25.343414 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.343272    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"4c065f7b4bbedaf889c3cd94a0aaa06f79b6e1bdd95892f83308c46850009e07"}
May 28 00:09:25.343414 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:25.343293    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"5e59f6b95c0a8c7b21f78903f4f16da979149acb1923de3755a034e425fbcefd"}
May 28 00:09:29.885560 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:29.885515    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused"
May 28 00:09:29.917071 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:29.917039    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 00:09:31.190239 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:31.190181    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 28 00:09:32.159910 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:32.159868    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 00:09:39.886124 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:39.886075    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused"
May 28 00:09:39.916988 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:39.916953    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 00:09:41.191098 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:41.191048    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 28 00:09:42.160844 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:42.160805    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 00:09:49.885256 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:49.885194    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused"
May 28 00:09:49.885709 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:49.885372    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:09:49.917858 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:49.917823    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 00:09:49.918029 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:49.917920    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:09:51.190441 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:51.190392    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 28 00:09:51.190869 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:51.190529    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:09:52.160716 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:52.160665    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 00:09:52.160940 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:52.160821    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:09:54.427853 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.427766    2572 generic.go:358] "Generic (PLEG): container finished" podID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerID="a1d0d4fa77fbd1c9edf82acd04db499ab5485e6f2510c8d2e11a602eeecdbd2a" exitCode=137
May 28 00:09:54.427853 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.427807    2572 generic.go:358] "Generic (PLEG): container finished" podID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerID="5e7d8c4000faeec473f5fc41bcf96bb0e9a27eae87aff303265d51d0ad732842" exitCode=137
May 28 00:09:54.428432 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.427941    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"a1d0d4fa77fbd1c9edf82acd04db499ab5485e6f2510c8d2e11a602eeecdbd2a"}
May 28 00:09:54.428432 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.427973    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"5e7d8c4000faeec473f5fc41bcf96bb0e9a27eae87aff303265d51d0ad732842"}
May 28 00:09:54.430928 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.430905    2572 generic.go:358] "Generic (PLEG): container finished" podID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerID="97e0e306296c7309e6bace76f750f1b98109d53a6a8bdb81e3502c6b262a0c8c" exitCode=137
May 28 00:09:54.431045 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.430956    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerDied","Data":"97e0e306296c7309e6bace76f750f1b98109d53a6a8bdb81e3502c6b262a0c8c"}
May 28 00:09:54.488470 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.488444    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:09:54.674467 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.674445    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:09:54.675624 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675602    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-ui-tls\") pod \"687a3b3a-eb87-4844-987a-b941fec5edf4\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") "
May 28 00:09:54.675726 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675639    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-online-tls\") pod \"687a3b3a-eb87-4844-987a-b941fec5edf4\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") "
May 28 00:09:54.675726 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675666    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2wh5\" (UniqueName: \"kubernetes.io/projected/687a3b3a-eb87-4844-987a-b941fec5edf4-kube-api-access-l2wh5\") pod \"687a3b3a-eb87-4844-987a-b941fec5edf4\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") "
May 28 00:09:54.675726 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675703    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/687a3b3a-eb87-4844-987a-b941fec5edf4-registry-tls\") pod \"687a3b3a-eb87-4844-987a-b941fec5edf4\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") "
May 28 00:09:54.675841 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675775    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-ui-tls\") pod \"98c0a07e-96cf-4aa0-a610-322683d457f0\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") "
May 28 00:09:54.675841 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675804    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/687a3b3a-eb87-4844-987a-b941fec5edf4-feast-data\") pod \"687a3b3a-eb87-4844-987a-b941fec5edf4\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") "
May 28 00:09:54.675841 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675829    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-offline-tls\") pod \"687a3b3a-eb87-4844-987a-b941fec5edf4\" (UID: \"687a3b3a-eb87-4844-987a-b941fec5edf4\") "
May 28 00:09:54.675980 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675865    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/98c0a07e-96cf-4aa0-a610-322683d457f0-feast-data\") pod \"98c0a07e-96cf-4aa0-a610-322683d457f0\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") "
May 28 00:09:54.675980 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.675905    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-registry-tls\") pod \"98c0a07e-96cf-4aa0-a610-322683d457f0\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") "
May 28 00:09:54.676206 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.676123    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687a3b3a-eb87-4844-987a-b941fec5edf4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "687a3b3a-eb87-4844-987a-b941fec5edf4" (UID: "687a3b3a-eb87-4844-987a-b941fec5edf4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:09:54.676783 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.676740    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687a3b3a-eb87-4844-987a-b941fec5edf4-feast-data" (OuterVolumeSpecName: "feast-data") pod "687a3b3a-eb87-4844-987a-b941fec5edf4" (UID: "687a3b3a-eb87-4844-987a-b941fec5edf4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:09:54.677006 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.676967    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c0a07e-96cf-4aa0-a610-322683d457f0-feast-data" (OuterVolumeSpecName: "feast-data") pod "98c0a07e-96cf-4aa0-a610-322683d457f0" (UID: "98c0a07e-96cf-4aa0-a610-322683d457f0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:09:54.678194 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.678169    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "98c0a07e-96cf-4aa0-a610-322683d457f0" (UID: "98c0a07e-96cf-4aa0-a610-322683d457f0"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.678401 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.678335    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "687a3b3a-eb87-4844-987a-b941fec5edf4" (UID: "687a3b3a-eb87-4844-987a-b941fec5edf4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.678644 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.678625    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687a3b3a-eb87-4844-987a-b941fec5edf4-kube-api-access-l2wh5" (OuterVolumeSpecName: "kube-api-access-l2wh5") pod "687a3b3a-eb87-4844-987a-b941fec5edf4" (UID: "687a3b3a-eb87-4844-987a-b941fec5edf4"). InnerVolumeSpecName "kube-api-access-l2wh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:09:54.678760 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.678731    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "98c0a07e-96cf-4aa0-a610-322683d457f0" (UID: "98c0a07e-96cf-4aa0-a610-322683d457f0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.678811 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.678755    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "687a3b3a-eb87-4844-987a-b941fec5edf4" (UID: "687a3b3a-eb87-4844-987a-b941fec5edf4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.678853 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.678834    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-online-tls" (OuterVolumeSpecName: "online-tls") pod "687a3b3a-eb87-4844-987a-b941fec5edf4" (UID: "687a3b3a-eb87-4844-987a-b941fec5edf4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.776957 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.776931    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92m7t\" (UniqueName: \"kubernetes.io/projected/98c0a07e-96cf-4aa0-a610-322683d457f0-kube-api-access-92m7t\") pod \"98c0a07e-96cf-4aa0-a610-322683d457f0\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") "
May 28 00:09:54.777120 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.776970    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-offline-tls\") pod \"98c0a07e-96cf-4aa0-a610-322683d457f0\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") "
May 28 00:09:54.777120 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777060    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-online-tls\") pod \"98c0a07e-96cf-4aa0-a610-322683d457f0\" (UID: \"98c0a07e-96cf-4aa0-a610-322683d457f0\") "
May 28 00:09:54.777251 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777235    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777259    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wh5\" (UniqueName: \"kubernetes.io/projected/687a3b3a-eb87-4844-987a-b941fec5edf4-kube-api-access-l2wh5\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777274    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/687a3b3a-eb87-4844-987a-b941fec5edf4-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777289    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777339 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777302    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/687a3b3a-eb87-4844-987a-b941fec5edf4-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777503 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777342    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777503 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777354    2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/98c0a07e-96cf-4aa0-a610-322683d457f0-feast-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777503 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777368    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.777503 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.777380    2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/687a3b3a-eb87-4844-987a-b941fec5edf4-ui-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.779085 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.779058    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "98c0a07e-96cf-4aa0-a610-322683d457f0" (UID: "98c0a07e-96cf-4aa0-a610-322683d457f0"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.779172 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.779086    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-online-tls" (OuterVolumeSpecName: "online-tls") pod "98c0a07e-96cf-4aa0-a610-322683d457f0" (UID: "98c0a07e-96cf-4aa0-a610-322683d457f0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:09:54.779172 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.779105    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c0a07e-96cf-4aa0-a610-322683d457f0-kube-api-access-92m7t" (OuterVolumeSpecName: "kube-api-access-92m7t") pod "98c0a07e-96cf-4aa0-a610-322683d457f0" (UID: "98c0a07e-96cf-4aa0-a610-322683d457f0"). InnerVolumeSpecName "kube-api-access-92m7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:09:54.878558 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.878535    2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-online-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.878558 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.878558    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92m7t\" (UniqueName: \"kubernetes.io/projected/98c0a07e-96cf-4aa0-a610-322683d457f0-kube-api-access-92m7t\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:54.878675 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:54.878568    2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/98c0a07e-96cf-4aa0-a610-322683d457f0-offline-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:09:55.436487 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.436456    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"
May 28 00:09:55.436487 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.436472    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67" event={"ID":"98c0a07e-96cf-4aa0-a610-322683d457f0","Type":"ContainerDied","Data":"0cebd2f64d4925846870450cc269c84b07b3342d4206888d61cae04c31aa16fa"}
May 28 00:09:55.436985 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.436510    2572 scope.go:117] "RemoveContainer" containerID="4c065f7b4bbedaf889c3cd94a0aaa06f79b6e1bdd95892f83308c46850009e07"
May 28 00:09:55.438955 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.438929    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf" event={"ID":"687a3b3a-eb87-4844-987a-b941fec5edf4","Type":"ContainerDied","Data":"d92e0e0a189201c29719c65e0cb7b36ec728add1ddd03931c0e39c6d6f35e705"}
May 28 00:09:55.439133 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.439035    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"
May 28 00:09:55.449879 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.449861    2572 scope.go:117] "RemoveContainer" containerID="a1d0d4fa77fbd1c9edf82acd04db499ab5485e6f2510c8d2e11a602eeecdbd2a"
May 28 00:09:55.457110 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.457086    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"]
May 28 00:09:55.457283 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.457227    2572 scope.go:117] "RemoveContainer" containerID="5e59f6b95c0a8c7b21f78903f4f16da979149acb1923de3755a034e425fbcefd"
May 28 00:09:55.462980 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.462956    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-x6r67"]
May 28 00:09:55.465204 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.465185    2572 scope.go:117] "RemoveContainer" containerID="5e7d8c4000faeec473f5fc41bcf96bb0e9a27eae87aff303265d51d0ad732842"
May 28 00:09:55.471779 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.471765    2572 scope.go:117] "RemoveContainer" containerID="27efc2e69f67b6e5b707b3d6e7b33176ea2c816476296fe478af2adab158a35f"
May 28 00:09:55.476184 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.476165    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"]
May 28 00:09:55.479384 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.479334    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-pm8wf"]
May 28 00:09:55.479907 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.479891    2572 scope.go:117] "RemoveContainer" containerID="0c9abce127c8d8a6e824260f63ed844ef5dfad6f0248e735a2dfe6229b05d7e0"
May 28 00:09:55.486569 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.486555    2572 scope.go:117] "RemoveContainer" containerID="ea583a415d3e7f3995f47b8f955b4b5ad2b82f5ccf646bce80e311f11ffecbc8"
May 28 00:09:55.493382 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.493362    2572 scope.go:117] "RemoveContainer" containerID="97e0e306296c7309e6bace76f750f1b98109d53a6a8bdb81e3502c6b262a0c8c"
May 28 00:09:55.499843 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.499827    2572 scope.go:117] "RemoveContainer" containerID="5d7544f6f37007509b64d1de6b340bc65eab2cab91d7c786359ce4bdc45b3dad"
May 28 00:09:55.506139 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.506124    2572 scope.go:117] "RemoveContainer" containerID="f7fbea439660103b03ed9d56d80de2ac22cfa843902da0693f1cbd22fcadd0f6"
May 28 00:09:55.512683 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:55.512669    2572 scope.go:117] "RemoveContainer" containerID="ded74cb79cc342aee9acc90eb7ebec78bdb9640aa39c98e3120d726c118b20af"
May 28 00:09:57.219496 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:57.219464    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" path="/var/lib/kubelet/pods/687a3b3a-eb87-4844-987a-b941fec5edf4/volumes"
May 28 00:09:57.220193 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:09:57.220170    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" path="/var/lib/kubelet/pods/98c0a07e-96cf-4aa0-a610-322683d457f0/volumes"
May 28 00:10:13.409203 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409163    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m4pnl/must-gather-nqff2"]
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409491    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="feast-init"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409504    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="feast-init"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409512    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409517    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409526    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409532    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409540    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409545    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409553    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="feast-init"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409558    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="feast-init"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409565    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="registry"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409570    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="registry"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409576    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409581    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409586    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="feast-apply"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409590    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="feast-apply"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409595    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="offline"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409600    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="offline"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409608    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="feast-apply"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409615    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="feast-apply"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="offline"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409625    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="offline"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409667    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="offline"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409676    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="ui"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409683    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="registry"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409688    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="offline"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409694    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="online"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409702    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="98c0a07e-96cf-4aa0-a610-322683d457f0" containerName="online"
May 28 00:10:13.409704 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.409708    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="687a3b3a-eb87-4844-987a-b941fec5edf4" containerName="ui"
May 28 00:10:13.412444 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.412426    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.414861 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.414836    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4pnl\"/\"kube-root-ca.crt\""
May 28 00:10:13.414861 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.414846    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4pnl\"/\"openshift-service-ca.crt\""
May 28 00:10:13.415022 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.414936    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m4pnl\"/\"default-dockercfg-zsvhn\""
May 28 00:10:13.419494 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.419139    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4pnl/must-gather-nqff2"]
May 28 00:10:13.515381 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.515356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgr4\" (UniqueName: \"kubernetes.io/projected/b1a478f4-fdbb-49c0-82c0-04cd357aac77-kube-api-access-6mgr4\") pod \"must-gather-nqff2\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") " pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.515541 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.515390    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1a478f4-fdbb-49c0-82c0-04cd357aac77-must-gather-output\") pod \"must-gather-nqff2\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") " pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.616564 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.616528    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgr4\" (UniqueName: \"kubernetes.io/projected/b1a478f4-fdbb-49c0-82c0-04cd357aac77-kube-api-access-6mgr4\") pod \"must-gather-nqff2\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") " pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.616735 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.616570    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1a478f4-fdbb-49c0-82c0-04cd357aac77-must-gather-output\") pod \"must-gather-nqff2\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") " pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.616912 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.616891    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1a478f4-fdbb-49c0-82c0-04cd357aac77-must-gather-output\") pod \"must-gather-nqff2\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") " pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.624790 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.624758    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgr4\" (UniqueName: \"kubernetes.io/projected/b1a478f4-fdbb-49c0-82c0-04cd357aac77-kube-api-access-6mgr4\") pod \"must-gather-nqff2\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") " pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.736887 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.736813    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:13.859106 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:13.859053    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4pnl/must-gather-nqff2"]
May 28 00:10:13.862068 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:10:13.862039    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1a478f4_fdbb_49c0_82c0_04cd357aac77.slice/crio-7a96d423f1c045cb068a92f497cdb877d9817a88f9d5155e975e7b41b4461a6b WatchSource:0}: Error finding container 7a96d423f1c045cb068a92f497cdb877d9817a88f9d5155e975e7b41b4461a6b: Status 404 returned error can't find the container with id 7a96d423f1c045cb068a92f497cdb877d9817a88f9d5155e975e7b41b4461a6b
May 28 00:10:14.495284 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:14.495244    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4pnl/must-gather-nqff2" event={"ID":"b1a478f4-fdbb-49c0-82c0-04cd357aac77","Type":"ContainerStarted","Data":"7a96d423f1c045cb068a92f497cdb877d9817a88f9d5155e975e7b41b4461a6b"}
May 28 00:10:20.515382 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:20.515263    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4pnl/must-gather-nqff2" event={"ID":"b1a478f4-fdbb-49c0-82c0-04cd357aac77","Type":"ContainerStarted","Data":"446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f"}
May 28 00:10:20.515382 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:20.515324    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4pnl/must-gather-nqff2" event={"ID":"b1a478f4-fdbb-49c0-82c0-04cd357aac77","Type":"ContainerStarted","Data":"1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"}
May 28 00:10:20.533232 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:20.533168    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-m4pnl/must-gather-nqff2" podStartSLOduration=1.4717178 podStartE2EDuration="7.533149379s" podCreationTimestamp="2026-05-28 00:10:13 +0000 UTC" firstStartedPulling="2026-05-28 00:10:13.86399728 +0000 UTC m=+1763.253501071" lastFinishedPulling="2026-05-28 00:10:19.925428857 +0000 UTC m=+1769.314932650" observedRunningTime="2026-05-28 00:10:20.53248019 +0000 UTC m=+1769.921984005" watchObservedRunningTime="2026-05-28 00:10:20.533149379 +0000 UTC m=+1769.922653194"
May 28 00:10:29.541908 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:29.541872    2572 generic.go:358] "Generic (PLEG): container finished" podID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerID="1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6" exitCode=0
May 28 00:10:29.542295 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:29.541939    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4pnl/must-gather-nqff2" event={"ID":"b1a478f4-fdbb-49c0-82c0-04cd357aac77","Type":"ContainerDied","Data":"1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"}
May 28 00:10:29.542295 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:29.542243    2572 scope.go:117] "RemoveContainer" containerID="1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"
May 28 00:10:30.502144 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:30.502107    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m4pnl_must-gather-nqff2_b1a478f4-fdbb-49c0-82c0-04cd357aac77/gather/0.log"
May 28 00:10:31.082858 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.082776    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lf7dz/must-gather-8phhm"]
May 28 00:10:31.084923 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.084906    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.087681 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.087658    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lf7dz\"/\"kube-root-ca.crt\""
May 28 00:10:31.088440 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.088423    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lf7dz\"/\"openshift-service-ca.crt\""
May 28 00:10:31.088570 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.088466    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lf7dz\"/\"default-dockercfg-mjg96\""
May 28 00:10:31.094163 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.094140    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lf7dz/must-gather-8phhm"]
May 28 00:10:31.158629 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.158594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/47eb66df-3ce9-4bfd-81b0-926e2cc441b4-must-gather-output\") pod \"must-gather-8phhm\" (UID: \"47eb66df-3ce9-4bfd-81b0-926e2cc441b4\") " pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.158813 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.158650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnkr\" (UniqueName: \"kubernetes.io/projected/47eb66df-3ce9-4bfd-81b0-926e2cc441b4-kube-api-access-wvnkr\") pod \"must-gather-8phhm\" (UID: \"47eb66df-3ce9-4bfd-81b0-926e2cc441b4\") " pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.259657 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.259626    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/47eb66df-3ce9-4bfd-81b0-926e2cc441b4-must-gather-output\") pod \"must-gather-8phhm\" (UID: \"47eb66df-3ce9-4bfd-81b0-926e2cc441b4\") " pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.259813 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.259691    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnkr\" (UniqueName: \"kubernetes.io/projected/47eb66df-3ce9-4bfd-81b0-926e2cc441b4-kube-api-access-wvnkr\") pod \"must-gather-8phhm\" (UID: \"47eb66df-3ce9-4bfd-81b0-926e2cc441b4\") " pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.259954 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.259934    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/47eb66df-3ce9-4bfd-81b0-926e2cc441b4-must-gather-output\") pod \"must-gather-8phhm\" (UID: \"47eb66df-3ce9-4bfd-81b0-926e2cc441b4\") " pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.268362 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.268334    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnkr\" (UniqueName: \"kubernetes.io/projected/47eb66df-3ce9-4bfd-81b0-926e2cc441b4-kube-api-access-wvnkr\") pod \"must-gather-8phhm\" (UID: \"47eb66df-3ce9-4bfd-81b0-926e2cc441b4\") " pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.394972 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.394864    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf7dz/must-gather-8phhm"
May 28 00:10:31.520266 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.520203    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lf7dz/must-gather-8phhm"]
May 28 00:10:31.522883 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:10:31.522858    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47eb66df_3ce9_4bfd_81b0_926e2cc441b4.slice/crio-f0bfc75e3a53051a2e2a0e523d4e5ff18ba76f7797bd3b7cb18ba0c3e82d78a3 WatchSource:0}: Error finding container f0bfc75e3a53051a2e2a0e523d4e5ff18ba76f7797bd3b7cb18ba0c3e82d78a3: Status 404 returned error can't find the container with id f0bfc75e3a53051a2e2a0e523d4e5ff18ba76f7797bd3b7cb18ba0c3e82d78a3
May 28 00:10:31.524473 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.524455    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:10:31.548871 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:31.548840    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf7dz/must-gather-8phhm" event={"ID":"47eb66df-3ce9-4bfd-81b0-926e2cc441b4","Type":"ContainerStarted","Data":"f0bfc75e3a53051a2e2a0e523d4e5ff18ba76f7797bd3b7cb18ba0c3e82d78a3"}
May 28 00:10:32.555230 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:32.555137    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf7dz/must-gather-8phhm" event={"ID":"47eb66df-3ce9-4bfd-81b0-926e2cc441b4","Type":"ContainerStarted","Data":"6be839439cf132ee835ad36f36d899a8cd640634297b2a65c9137f2f5419b984"}
May 28 00:10:32.555230 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:32.555177    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf7dz/must-gather-8phhm" event={"ID":"47eb66df-3ce9-4bfd-81b0-926e2cc441b4","Type":"ContainerStarted","Data":"bca2dfb13b1aa678d16e59d3ea7936a1302c303ca9687ca6fd3355dc36649916"}
May 28 00:10:32.572070 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:32.572019    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lf7dz/must-gather-8phhm" podStartSLOduration=0.882740112 podStartE2EDuration="1.572005383s" podCreationTimestamp="2026-05-28 00:10:31 +0000 UTC" firstStartedPulling="2026-05-28 00:10:31.524613515 +0000 UTC m=+1780.914117307" lastFinishedPulling="2026-05-28 00:10:32.213878784 +0000 UTC m=+1781.603382578" observedRunningTime="2026-05-28 00:10:32.57057427 +0000 UTC m=+1781.960078085" watchObservedRunningTime="2026-05-28 00:10:32.572005383 +0000 UTC m=+1781.961509205"
May 28 00:10:33.503714 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:33.503682    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rkp2l_6554f633-c725-4f96-bb1c-f019f8185210/global-pull-secret-syncer/0.log"
May 28 00:10:33.636063 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:33.636023    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hzvkg_3d28d858-235d-4e96-9f9a-57634afaa5c9/konnectivity-agent/0.log"
May 28 00:10:33.718128 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:33.718092    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-162.ec2.internal_a329a27d84f7b98ef3b36fc9a735cfdf/haproxy/0.log"
May 28 00:10:35.923225 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:35.923180    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-m4pnl/must-gather-nqff2"]
May 28 00:10:35.923686 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:35.923494    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-m4pnl/must-gather-nqff2" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="copy" containerID="cri-o://446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f" gracePeriod=2
May 28 00:10:35.925710 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:35.925663    2572 status_manager.go:895] "Failed to get status for pod" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" pod="openshift-must-gather-m4pnl/must-gather-nqff2" err="pods \"must-gather-nqff2\" is forbidden: User \"system:node:ip-10-0-133-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m4pnl\": no relationship found between node 'ip-10-0-133-162.ec2.internal' and this object"
May 28 00:10:35.927848 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:35.927822    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-m4pnl/must-gather-nqff2"]
May 28 00:10:36.280116 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.280064    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m4pnl_must-gather-nqff2_b1a478f4-fdbb-49c0-82c0-04cd357aac77/copy/0.log"
May 28 00:10:36.280852 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.280535    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:36.284586 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.284546    2572 status_manager.go:895] "Failed to get status for pod" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" pod="openshift-must-gather-m4pnl/must-gather-nqff2" err="pods \"must-gather-nqff2\" is forbidden: User \"system:node:ip-10-0-133-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m4pnl\": no relationship found between node 'ip-10-0-133-162.ec2.internal' and this object"
May 28 00:10:36.411982 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.411344    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1a478f4-fdbb-49c0-82c0-04cd357aac77-must-gather-output\") pod \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") "
May 28 00:10:36.411982 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.411401    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mgr4\" (UniqueName: \"kubernetes.io/projected/b1a478f4-fdbb-49c0-82c0-04cd357aac77-kube-api-access-6mgr4\") pod \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\" (UID: \"b1a478f4-fdbb-49c0-82c0-04cd357aac77\") "
May 28 00:10:36.412682 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.412645    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a478f4-fdbb-49c0-82c0-04cd357aac77-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b1a478f4-fdbb-49c0-82c0-04cd357aac77" (UID: "b1a478f4-fdbb-49c0-82c0-04cd357aac77"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:10:36.414630 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.414570    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a478f4-fdbb-49c0-82c0-04cd357aac77-kube-api-access-6mgr4" (OuterVolumeSpecName: "kube-api-access-6mgr4") pod "b1a478f4-fdbb-49c0-82c0-04cd357aac77" (UID: "b1a478f4-fdbb-49c0-82c0-04cd357aac77"). InnerVolumeSpecName "kube-api-access-6mgr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:10:36.512176 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.512062    2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1a478f4-fdbb-49c0-82c0-04cd357aac77-must-gather-output\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:10:36.512176 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.512101    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mgr4\" (UniqueName: \"kubernetes.io/projected/b1a478f4-fdbb-49c0-82c0-04cd357aac77-kube-api-access-6mgr4\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\""
May 28 00:10:36.576949 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.576869    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m4pnl_must-gather-nqff2_b1a478f4-fdbb-49c0-82c0-04cd357aac77/copy/0.log"
May 28 00:10:36.579834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.577543    2572 generic.go:358] "Generic (PLEG): container finished" podID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerID="446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f" exitCode=143
May 28 00:10:36.579834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.577613    2572 scope.go:117] "RemoveContainer" containerID="446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f"
May 28 00:10:36.579834 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.577748    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4pnl/must-gather-nqff2"
May 28 00:10:36.582193 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.582158    2572 status_manager.go:895] "Failed to get status for pod" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" pod="openshift-must-gather-m4pnl/must-gather-nqff2" err="pods \"must-gather-nqff2\" is forbidden: User \"system:node:ip-10-0-133-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m4pnl\": no relationship found between node 'ip-10-0-133-162.ec2.internal' and this object"
May 28 00:10:36.593210 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.591402    2572 scope.go:117] "RemoveContainer" containerID="1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"
May 28 00:10:36.595287 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.595257    2572 status_manager.go:895] "Failed to get status for pod" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" pod="openshift-must-gather-m4pnl/must-gather-nqff2" err="pods \"must-gather-nqff2\" is forbidden: User \"system:node:ip-10-0-133-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m4pnl\": no relationship found between node 'ip-10-0-133-162.ec2.internal' and this object"
May 28 00:10:36.612554 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.612527    2572 scope.go:117] "RemoveContainer" containerID="446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f"
May 28 00:10:36.613209 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:10:36.613179    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f\": container with ID starting with 446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f not found: ID does not exist" containerID="446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f"
May 28 00:10:36.613431 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.613400    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f"} err="failed to get container status \"446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f\": rpc error: code = NotFound desc = could not find container \"446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f\": container with ID starting with 446d50ef78a20b06b129b66ff928da1af171d24d7419fb28c9bea02433a9602f not found: ID does not exist"
May 28 00:10:36.613561 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.613548    2572 scope.go:117] "RemoveContainer" containerID="1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"
May 28 00:10:36.614010 ip-10-0-133-162 kubenswrapper[2572]: E0528 00:10:36.613980    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6\": container with ID starting with 1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6 not found: ID does not exist" containerID="1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"
May 28 00:10:36.614128 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:36.614016    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6"} err="failed to get container status \"1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6\": rpc error: code = NotFound desc = could not find container \"1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6\": container with ID starting with 1d31545ebe75cf38d055c7ec2de4dc8798574a54984e59633545ad4c52e377f6 not found: ID does not exist"
May 28 00:10:37.221165 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.221131    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" path="/var/lib/kubelet/pods/b1a478f4-fdbb-49c0-82c0-04cd357aac77/volumes"
May 28 00:10:37.396431 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.396358    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f4857778d-szdtm_b1bbc942-d7f9-42c0-9f46-28de571bbcf7/metrics-server/0.log"
May 28 00:10:37.422624 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.422592    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-sfj7f_01319e4a-e3e5-4e1e-b0e5-b0efa5113687/monitoring-plugin/0.log"
May 28 00:10:37.597284 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.597203    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdnfv_fd52a86d-0131-4b50-9a21-480c11028692/node-exporter/0.log"
May 28 00:10:37.622604 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.622575    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdnfv_fd52a86d-0131-4b50-9a21-480c11028692/kube-rbac-proxy/0.log"
May 28 00:10:37.646430 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.646407    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdnfv_fd52a86d-0131-4b50-9a21-480c11028692/init-textfile/0.log"
May 28 00:10:37.978205 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:37.978159    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-9gddx_37a0dd1e-ed46-462c-8b1d-c8a55c448f4d/prometheus-operator/0.log"
May 28 00:10:38.005884 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:38.005858    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-9gddx_37a0dd1e-ed46-462c-8b1d-c8a55c448f4d/kube-rbac-proxy/0.log"
May 28 00:10:38.032580 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:38.032527    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-cjx69_43474b03-1e8a-4922-9b6d-c6b4c1291521/prometheus-operator-admission-webhook/0.log"
May 28 00:10:39.959644 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959609    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"]
May 28 00:10:39.960087 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959906    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="gather"
May 28 00:10:39.960087 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959916    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="gather"
May 28 00:10:39.960087 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959925    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="copy"
May 28 00:10:39.960087 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959930    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="copy"
May 28 00:10:39.960087 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959979    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="copy"
May 28 00:10:39.960087 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.959988    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1a478f4-fdbb-49c0-82c0-04cd357aac77" containerName="gather"
May 28 00:10:39.963004 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.962975    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:39.974016 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:39.973974    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"]
May 28 00:10:40.002233 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.002206    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f57667c89-mj9fr_102e367a-1618-4d29-b622-e29d99d617e7/console/0.log"
May 28 00:10:40.041900 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.041842    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-v28ps_d22b1e67-08e2-4858-871d-7a64ef7b09d5/download-server/0.log"
May 28 00:10:40.043185 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.043162    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-lib-modules\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.043289 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.043213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-sys\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.043359 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.043298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mw5h\" (UniqueName: \"kubernetes.io/projected/af5dedea-191e-42ba-b898-c7310586cf30-kube-api-access-2mw5h\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.043403 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.043374    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-proc\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.043403 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.043395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-podres\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144605 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144573    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-proc\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144605 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144608    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-podres\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144807 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144693    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-proc\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144807 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144731    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-podres\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144807 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144732    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-lib-modules\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144807 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144792    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-sys\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144965 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144816    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-lib-modules\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144965 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144855    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mw5h\" (UniqueName: \"kubernetes.io/projected/af5dedea-191e-42ba-b898-c7310586cf30-kube-api-access-2mw5h\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.144965 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.144863    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af5dedea-191e-42ba-b898-c7310586cf30-sys\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.152771 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.152748    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mw5h\" (UniqueName: \"kubernetes.io/projected/af5dedea-191e-42ba-b898-c7310586cf30-kube-api-access-2mw5h\") pod \"perf-node-gather-daemonset-68vl2\" (UID: \"af5dedea-191e-42ba-b898-c7310586cf30\") " pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.276255 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.276174    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:40.422129 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.422094    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"]
May 28 00:10:40.425108 ip-10-0-133-162 kubenswrapper[2572]: W0528 00:10:40.425078    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaf5dedea_191e_42ba_b898_c7310586cf30.slice/crio-52b52e91ce2082a357517b7cd9c06f0f622d7fa8f19ef749f4e0761fccacd8bc WatchSource:0}: Error finding container 52b52e91ce2082a357517b7cd9c06f0f622d7fa8f19ef749f4e0761fccacd8bc: Status 404 returned error can't find the container with id 52b52e91ce2082a357517b7cd9c06f0f622d7fa8f19ef749f4e0761fccacd8bc
May 28 00:10:40.594053 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.593973    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2" event={"ID":"af5dedea-191e-42ba-b898-c7310586cf30","Type":"ContainerStarted","Data":"fccabfe20a9037225db022d45a7b98444e45e46ba1d672bea757fa97fad0b7e2"}
May 28 00:10:40.594053 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.594012    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2" event={"ID":"af5dedea-191e-42ba-b898-c7310586cf30","Type":"ContainerStarted","Data":"52b52e91ce2082a357517b7cd9c06f0f622d7fa8f19ef749f4e0761fccacd8bc"}
May 28 00:10:40.594284 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:40.594067    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:41.033878 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:41.033845    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44grr_b39519dd-a77f-4429-913c-18762f94546d/dns/0.log"
May 28 00:10:41.055936 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:41.055908    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44grr_b39519dd-a77f-4429-913c-18762f94546d/kube-rbac-proxy/0.log"
May 28 00:10:41.192571 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:41.192543    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flbbx_8b3785e5-1e3a-4306-b7cb-64acde7e65ed/dns-node-resolver/0.log"
May 28 00:10:41.620058 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:41.620026    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d555f9d9c-vcntc_77242b73-dafd-43f9-b91c-ca07a5a7f56a/registry/0.log"
May 28 00:10:41.664976 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:41.664941    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lw89q_0616e250-c245-4aa5-aaaa-ae15d9e3234e/node-ca/0.log"
May 28 00:10:42.627284 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:42.627228    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-22265_30d6a16f-881b-4b68-892c-e306fd38659b/serve-healthcheck-canary/0.log"
May 28 00:10:43.128442 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:43.128412    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6jgs_7877bede-c140-4014-bbd1-c9d71fb5728f/kube-rbac-proxy/0.log"
May 28 00:10:43.146853 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:43.146830    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6jgs_7877bede-c140-4014-bbd1-c9d71fb5728f/exporter/0.log"
May 28 00:10:43.166176 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:43.166141    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6jgs_7877bede-c140-4014-bbd1-c9d71fb5728f/extractor/0.log"
May 28 00:10:46.607672 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:46.607643    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2"
May 28 00:10:46.625745 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:46.625688    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lf7dz/perf-node-gather-daemonset-68vl2" podStartSLOduration=7.625670695 podStartE2EDuration="7.625670695s" podCreationTimestamp="2026-05-28 00:10:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:10:40.61065289 +0000 UTC m=+1790.000156705" watchObservedRunningTime="2026-05-28 00:10:46.625670695 +0000 UTC m=+1796.015174510"
May 28 00:10:47.477473 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:47.477423    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ddqvm_4717a5f4-dc35-49e8-a0f1-b2e008417733/kube-storage-version-migrator-operator/1.log"
May 28 00:10:47.479321 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:47.479273    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ddqvm_4717a5f4-dc35-49e8-a0f1-b2e008417733/kube-storage-version-migrator-operator/0.log"
May 28 00:10:48.514520 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.514454    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/kube-multus-additional-cni-plugins/0.log"
May 28 00:10:48.538026 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.537995    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/egress-router-binary-copy/0.log"
May 28 00:10:48.562016 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.561979    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/cni-plugins/0.log"
May 28 00:10:48.583881 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.583847    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/bond-cni-plugin/0.log"
May 28 00:10:48.605037 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.605009    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/routeoverride-cni/0.log"
May 28 00:10:48.625774 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.625743    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/whereabouts-cni-bincopy/0.log"
May 28 00:10:48.646904 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.646879    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzrfc_8ff4dcda-d905-444a-88cc-49ab1ce8b7bf/whereabouts-cni/0.log"
May 28 00:10:48.838965 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.838864    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pjwfd_69ebaa5a-f129-4be0-b4a7-3dc11eafd58c/kube-multus/0.log"
May 28 00:10:48.917973 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.917945    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bp6lz_80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f/network-metrics-daemon/0.log"
May 28 00:10:48.944606 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:48.944578    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bp6lz_80a7eeb9-cd7a-4d9e-b82e-7805d21ddb3f/kube-rbac-proxy/0.log"
May 28 00:10:50.176920 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.176883    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-controller/0.log"
May 28 00:10:50.195125 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.195098    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:10:50.211808 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.211783    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/1.log"
May 28 00:10:50.236941 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.236910    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/kube-rbac-proxy-node/0.log"
May 28 00:10:50.262345 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.262288    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/kube-rbac-proxy-ovn-metrics/0.log"
May 28 00:10:50.281094 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.281066    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/northd/0.log"
May 28 00:10:50.304377 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.304343    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/nbdb/0.log"
May 28 00:10:50.327612 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.327581    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/sbdb/0.log"
May 28 00:10:50.515936 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:50.515861    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovnkube-controller/0.log"
May 28 00:10:51.236676 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:51.236599    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:10:51.237111 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:51.236878    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cpqgv_6fe3b412-20be-435b-9aaa-ef0602a81cd5/ovn-acl-logging/0.log"
May 28 00:10:51.761615 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:51.761585    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ttrwc_cd519bd5-eacb-4a30-a0ec-6e554d8001da/network-check-target-container/0.log"
May 28 00:10:52.603810 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:52.603781    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-n58tk_0cdb4626-9749-4c04-9159-2c63485fe159/iptables-alerter/0.log"
May 28 00:10:53.201859 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:53.201823    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4wvc2_b2bba202-befe-4b9a-9ca9-88e58a64a498/tuned/0.log"
May 28 00:10:54.833975 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:54.833939    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-64b5x_fddfa466-8322-4bb5-a94f-ad100bb45c81/cluster-samples-operator/0.log"
May 28 00:10:54.850317 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:54.850275    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-64b5x_fddfa466-8322-4bb5-a94f-ad100bb45c81/cluster-samples-operator-watch/0.log"
May 28 00:10:55.758696 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:55.758573    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-fhtrw_d5194cbd-67e6-43a3-8698-91ce60449345/service-ca-operator/1.log"
May 28 00:10:55.760301 ip-10-0-133-162 kubenswrapper[2572]: I0528 00:10:55.760242    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-fhtrw_d5194cbd-67e6-43a3-8698-91ce60449345/service-ca-operator/0.log"