Jul 07 18:20:38.394614 ip-10-0-130-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 18:20:38.394626 ip-10-0-130-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 18:20:38.394636 ip-10-0-130-31 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 18:20:38.394955 ip-10-0-130-31 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 18:20:48.593526 ip-10-0-130-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 18:20:48.593544 ip-10-0-130-31 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a2c8ef1f5fdc43e896fb7c1ee4f0760b --
Jul 07 18:22:50.763630 ip-10-0-130-31 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.123835    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128814    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128826    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128830    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128833    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128835    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128838    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128841    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 18:22:51.217110 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128843    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 18:22:51.194004 ip-10-0-130-31 systemd[1]: Started Kubernetes Kubelet.
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128846    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128849    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128852    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128854    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128857    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128860    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128862    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128865    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128868    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128870    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128873    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128875    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128878    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128884    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128887    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128889    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128893    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128895    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 18:22:51.218725 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128898    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128901    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128903    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128906    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128908    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128911    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128914    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128920    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128924    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128927    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128930    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128933    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128936    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128938    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128941    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128943    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128946    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128949    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128952    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 18:22:51.224815 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128956    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128959    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128962    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128965    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128968    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128970    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128972    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128975    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128977    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128980    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128983    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128985    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128988    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128991    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128993    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128996    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.128998    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129000    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129003    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129005    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 18:22:51.225455 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129009    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129012    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129014    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129017    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129020    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129022    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129025    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129028    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129031    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129035    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129037    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129040    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129044    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129047    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129050    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129053    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129056    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129059    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129062    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129065    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 18:22:51.226079 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129067    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129070    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129072    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129075    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129077    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129080    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129083    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129085    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129088    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129090    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129094    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129097    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129099    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129102    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129817    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129824    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129827    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129831    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129835    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 18:22:51.226805 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129839    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129842    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129846    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129849    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129851    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129854    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129857    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129860    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129862    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129865    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129868    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129870    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129873    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129875    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129878    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129880    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129883    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129886    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129889    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129891    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 18:22:51.227689 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129894    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129899    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129902    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129905    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129908    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129911    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129914    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129917    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129919    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129922    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129925    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129927    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129930    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129933    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129935    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129938    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129940    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129943    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129946    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129948    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 18:22:51.228549 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129951    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129954    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129957    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129959    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129962    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129964    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129967    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129969    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129972    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129974    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129977    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129979    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129982    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129984    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129987    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129989    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129992    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129994    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129997    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.129999    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 18:22:51.229135 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130001    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130006    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130008    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130011    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130013    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130016    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130021    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130024    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130027    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130030    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130032    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130035    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130037    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130041    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130044    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130046    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130049    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130051    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130054    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130057    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 18:22:51.229818 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130059    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130062    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130064    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130067    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130071    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130074    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130077    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130080    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130083    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130086    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130088    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130091    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130093    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130099    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130463    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130468    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130471    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130473    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130477    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 18:22:51.230648 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130479    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130482    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130484    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130487    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130489    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130492    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130494    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130497    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130500    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130502    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130505    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130508    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130510    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130513    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130515    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130517    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130520    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130522    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130525    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130528    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 18:22:51.231269 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130530    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130533    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130536    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130538    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130541    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130544    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130548    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130551    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130553    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130556    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130558    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130561    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130564    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130566    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130569    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130572    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130576    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130580    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 18:22:51.232142 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130584    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130587    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130590    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130592    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130595    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130597    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130600    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130602    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130604    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130607    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130610    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130612    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130615    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130617    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130619    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130622    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130624    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130629    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130633    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 18:22:51.232873 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130636    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130640    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130643    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130646    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130648    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130651    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130654    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130656    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130659    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130662    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130664    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130668    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130670    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130673    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130675    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130677    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130680    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130682    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130685    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130687    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 18:22:51.233393 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130690    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130692    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130695    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130697    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130700    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130702    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130705    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130707    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130710    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130712    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130716    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130718    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130721    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130724    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130727    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130729    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.130732    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130800    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130807    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130813    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130817    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 18:22:51.234074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130822    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130826    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130830    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130834    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130837    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130840    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130843    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130846    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130849    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130852    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130855    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130858    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130861    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130864    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130868    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130871    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130874    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130877    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130880    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130883    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130886    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130890    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130894    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130897    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 18:22:51.234691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130901    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130904    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130907    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130910    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130914    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130917    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130920    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130923    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130926    2565 flags.go:64] FLAG: --enable-server="true"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130930    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130934    2565 flags.go:64] FLAG: --event-burst="100"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130937    2565 flags.go:64] FLAG: --event-qps="50"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130940    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130943    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130946    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130949    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130952    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130955    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130958    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130961    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130963    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130966    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130969    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130972    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130974    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 18:22:52.960509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130977    2565 flags.go:64] FLAG: --feature-gates=""
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130981    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130983    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130986    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130991    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130994    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.130997    2565 flags.go:64] FLAG: --help="false"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131001    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-130-31.ec2.internal"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131004    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131007    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131009    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131012    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131016    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131018    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131021    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131024    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131027    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131030    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131033    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131036    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131039    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131042    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131045    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131047    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 18:22:52.978157 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131050    2565 flags.go:64] FLAG: --lock-file=""
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131053    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131055    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131058    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131063    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131066    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131069    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131072    2565 flags.go:64] FLAG: --logging-format="text"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131074    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131077    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131080    2565 flags.go:64] FLAG: --manifest-url=""
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131083    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131088    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131092    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131096    2565 flags.go:64] FLAG: --max-pods="110"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131100    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131103    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131106    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131108    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131111    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131114    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131117    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131123    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131126    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131129    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 18:22:52.978992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131132    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131136    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131141    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131144    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131147    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131149    2565 flags.go:64] FLAG: --port="10250"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131153    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131155    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0600cef9c04177163"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131158    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131162    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131165    2565 flags.go:64] FLAG: --register-node="true"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131167    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131171    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131174    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131177    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131179    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131183    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131186    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131189    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131193    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131196    2565 flags.go:64] FLAG: --runonce="false"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131198    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131201    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131206    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131209    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 18:22:52.979863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131212    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131215    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131217    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131220    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131223    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131226    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131228    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131231    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131234    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131238    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131241    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131246    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131249    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131251    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131255    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131258    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131260    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131263    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131265    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131268    2565 flags.go:64] FLAG: --v="2"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131272    2565 flags.go:64] FLAG: --version="false"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131275    2565 flags.go:64] FLAG: --vmodule=""
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131279    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.131282    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131402    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 18:22:52.980594 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131407    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131411    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131414    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131417    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131420    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131424    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131427    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131430    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131432    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131435    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131437    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131440    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131442    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131445    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131447    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131450    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131452    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131455    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131458    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 18:22:52.981601 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131460    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131464    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131468    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131471    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131473    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131476    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131479    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131481    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131484    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131486    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131489    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131491    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131494    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131497    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131501    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131504    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131506    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131509    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131512    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 18:22:52.982167 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131515    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131517    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131520    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131522    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131525    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131528    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131530    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131533    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131536    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131539    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131543    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131546    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131549    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131552    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131555    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131557    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131560    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131563    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131565    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131568    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 18:22:53.153570 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131570    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131573    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131575    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131577    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131580    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131582    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131585    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131589    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131591    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131594    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131596    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131600    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131603    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131605    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131609    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131613    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131616    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131619    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131621    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 18:22:53.154565 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131624    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131627    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131629    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131632    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131634    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131637    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131640    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131642    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131645    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131647    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131650    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131652    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131654    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131657    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131659    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131662    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131664    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131667    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131669    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131671    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 18:22:53.155100 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.131676    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.132631    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.139068    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.139084    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.139876    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.139884    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139953    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139957    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139961    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139964    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139967    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139970    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139972    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139975    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 18:22:53.155718 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139978    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139982    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139985    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139988    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139991    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139994    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.139997    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140000    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140003    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140005    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140008    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140011    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140013    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140016    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140019    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140021    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140024    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140027    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140029    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 18:22:53.156205 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140032    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140035    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140039    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140042    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140044    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140047    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140050    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140053    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140055    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140058    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140060    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140063    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140065    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140068    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140070    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140073    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140076    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140079    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140081    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140084    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 18:22:53.156784 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140086    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140089    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140091    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140094    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140096    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140099    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140101    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140104    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140106    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140109    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140112    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140114    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140117    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140120    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140125    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140128    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140131    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140133    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140136    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140138    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 18:22:53.157348 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140141    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140143    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140146    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140149    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140151    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140154    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140156    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140159    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140162    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140164    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140168    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140172    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140175    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140178    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140180    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140183    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140185    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140188    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140190    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 18:22:53.157984 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140193    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140195    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140198    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140200    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140203    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140206    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140208    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140214    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140217    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140221    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140224    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140227    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140229    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.140234    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 18:22:53.158509 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140339    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140343    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140346    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140349    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140369    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140372    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140375    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140378    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140381    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140383    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140386    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140389    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140392    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140394    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140397    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140399    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140402    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140404    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140407    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140409    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 18:22:53.158958 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140412    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140414    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140417    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140419    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140423    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140428    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140434    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140437    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140440    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140443    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140445    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140448    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140450    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140453    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140455    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140458    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140460    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140463    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140466    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 18:22:53.159557 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140468    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140471    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140473    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140476    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140478    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140481    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140483    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140486    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140488    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140491    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140493    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140496    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140498    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140501    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140504    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140506    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140508    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140511    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140513    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 18:22:53.160120 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140519    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140523    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140526    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140529    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140532    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140535    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140538    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140540    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140543    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140546    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140549    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140551    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140554    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140556    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140559    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140562    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140564    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140566    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140569    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 18:22:53.160698 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140571    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140574    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140576    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140579    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140581    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140584    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140586    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140589    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140591    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140593    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140596    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140598    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140601    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140607    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140610    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140613    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140616    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140619    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140622    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140624    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 18:22:53.161246 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140627    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:51.140629    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.140634    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.140725    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.142679    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.143915    2565 server.go:1019] "Starting client certificate rotation"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.144015    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.144053    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.146094    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.148602    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.162294    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.164272    2565 log.go:25] "Validated CRI v1 image API"
Jul 07 18:22:53.161810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.165127    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 18:22:53.162188 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.170202    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa42079c-7ecd-42a7-89fb-fb2861905156:/dev/nvme0n1p4 e42cb515-53d5-4a06-86e0-58c9897cc2a4:/dev/nvme0n1p3]
Jul 07 18:22:53.162188 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.170217    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.174194    2565 manager.go:216] Machine: {Timestamp:2026-07-07 18:22:51.1737295 +0000 UTC m=+0.299597996 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087924 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d083ac6005f2eebbde20e5904b31d SystemUUID:ec2d083a-c600-5f2e-ebbd-e20e5904b31d BootID:a2c8ef1f-5fdc-43e8-96fb-7c1ee4f0760b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4a:9e:49:0d:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4a:9e:49:0d:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:d9:3e:28:d5:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.174287    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.174385    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.177274    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.177295    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.177990    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.178003    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.178010    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.178164    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.178680    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.179838    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182656    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182801    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182842    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182857    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182875    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 18:22:53.162355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182888    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.182899    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.183279    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.183299    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.185256    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.187194    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.187230    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188311    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188328    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188338    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188348    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188378    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188390    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188405    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188411    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188435    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188443    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188465    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.188472    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.190070    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.190078    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.193019    2565 server.go:1268] "Started kubelet"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.193135    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.193178    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.193674    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.194798    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.197379    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.197569    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.197609    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.197874    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.205055    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g8c2c"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.206924    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.209074    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.209372    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g8c2c"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.209511    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.209625    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210172    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210184    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210184    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210244    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210252    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210682    2565 factory.go:55] Registering systemd factory
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210697    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210899    2565 factory.go:153] Registering CRI-O factory
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210912    2565 factory.go:223] Registration of the crio container factory successfully
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210961    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.210999    2565 factory.go:103] Registering Raw factory
Jul 07 18:22:53.162868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.211014    2565 manager.go:1196] Started watching for new ooms in manager
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.210511    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-31.ec2.internal.18c01506c122ea88  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-31.ec2.internal,UID:ip-10-0-130-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-31.ec2.internal,},FirstTimestamp:2026-07-07 18:22:51.19299444 +0000 UTC m=+0.318862935,LastTimestamp:2026-07-07 18:22:51.19299444 +0000 UTC m=+0.318862935,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-31.ec2.internal,}"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.211445    2565 manager.go:349] Starting recovery of all containers
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.214581    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.217814    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-31.ec2.internal\" not found" node="ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.220386    2565 manager.go:354] Recovery completed
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.224862    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.226857    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.226903    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.226918    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.227459    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.227474    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.227490    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.227561    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.228263    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.228306    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.228319    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.234915    2565 policy_none.go:49] "None policy: Start"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.234927    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.234937    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.235922    2565 policy_none.go:47] "Start"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.278945    2565 manager.go:337] "Starting Device Plugin manager"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.279089    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.279101    2565 server.go:87] "Starting device plugin registration server"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.279298    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.279307    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.279407    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.279520    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.279528    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.280021    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.280069    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.379528    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.380392    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.380416    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.380426    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.380451    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:51.387050    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:51.387068    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-31.ec2.internal\": node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:52.992414    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.040055    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.041221    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.041256    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 18:22:53.164311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.041275    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.041309    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.043924    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.093091    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.142227    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"]
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.142293    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.143648    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.143671    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.143682    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.144799    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.144942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.144970    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.145501    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.145513    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.145527    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.145532    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.145540    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.145541    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.146787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.146808    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.147428    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.147451    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.147484    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeHasSufficientPID"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.162067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05b4edd690f8cd78d388db58e1fdf50f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal\" (UID: \"05b4edd690f8cd78d388db58e1fdf50f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.162113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b4edd690f8cd78d388db58e1fdf50f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal\" (UID: \"05b4edd690f8cd78d388db58e1fdf50f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.167398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.162128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd00b61865beafaa890243361850b718-config\") pod \"kube-apiserver-proxy-ip-10-0-130-31.ec2.internal\" (UID: \"bd00b61865beafaa890243361850b718\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.170065    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-31.ec2.internal\" not found" node="ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.174664    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-31.ec2.internal\" not found" node="ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.193821    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.262298    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05b4edd690f8cd78d388db58e1fdf50f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal\" (UID: \"05b4edd690f8cd78d388db58e1fdf50f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.262260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05b4edd690f8cd78d388db58e1fdf50f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal\" (UID: \"05b4edd690f8cd78d388db58e1fdf50f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.262356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b4edd690f8cd78d388db58e1fdf50f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal\" (UID: \"05b4edd690f8cd78d388db58e1fdf50f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.262373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd00b61865beafaa890243361850b718-config\") pod \"kube-apiserver-proxy-ip-10-0-130-31.ec2.internal\" (UID: \"bd00b61865beafaa890243361850b718\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.262409    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd00b61865beafaa890243361850b718-config\") pod \"kube-apiserver-proxy-ip-10-0-130-31.ec2.internal\" (UID: \"bd00b61865beafaa890243361850b718\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.262438    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b4edd690f8cd78d388db58e1fdf50f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal\" (UID: \"05b4edd690f8cd78d388db58e1fdf50f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.356525 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.294361    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.523826 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.394810    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.523826 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.474230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.523826 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.477928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:53.523826 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.495483    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.595939    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.620221    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.693578    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.693672    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.693718    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.693720    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 18:22:53.700335 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.696732    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:53.855290 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.761793    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:17:52 +0000 UTC" deadline="2028-02-05 10:34:12.469194491 +0000 UTC"
Jul 07 18:22:53.855290 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:53.761813    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13864h11m18.707383844s"
Jul 07 18:22:53.855290 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.797180    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:54.034669 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.897787    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:54.034669 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:53.998232    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:54.034669 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:54.027796    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b4edd690f8cd78d388db58e1fdf50f.slice/crio-0156f5f9a96fb6d98784bb355b4a7a27b79d490d9675f9abd320b00bcb983bbd WatchSource:0}: Error finding container 0156f5f9a96fb6d98784bb355b4a7a27b79d490d9675f9abd320b00bcb983bbd: Status 404 returned error can't find the container with id 0156f5f9a96fb6d98784bb355b4a7a27b79d490d9675f9abd320b00bcb983bbd
Jul 07 18:22:54.034669 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:54.028249    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd00b61865beafaa890243361850b718.slice/crio-167f88d47ca9b4ac2b13302da29ee8d7effc5d1f1131ac4dba692d1f658592b7 WatchSource:0}: Error finding container 167f88d47ca9b4ac2b13302da29ee8d7effc5d1f1131ac4dba692d1f658592b7: Status 404 returned error can't find the container with id 167f88d47ca9b4ac2b13302da29ee8d7effc5d1f1131ac4dba692d1f658592b7
Jul 07 18:22:54.034669 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.032877    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:22:54.198980 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.044447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal" event={"ID":"05b4edd690f8cd78d388db58e1fdf50f","Type":"ContainerStarted","Data":"0156f5f9a96fb6d98784bb355b4a7a27b79d490d9675f9abd320b00bcb983bbd"}
Jul 07 18:22:54.198980 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.045523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal" event={"ID":"bd00b61865beafaa890243361850b718","Type":"ContainerStarted","Data":"167f88d47ca9b4ac2b13302da29ee8d7effc5d1f1131ac4dba692d1f658592b7"}
Jul 07 18:22:54.198980 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.098941    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:54.199733 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.199620    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:54.379663 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.300326    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-31.ec2.internal\" not found"
Jul 07 18:22:54.379663 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.300698    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:54.379663 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.364210    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:54.379663 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.379483    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 18:22:54.380511 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.380367    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal"
Jul 07 18:22:54.542016 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.392310    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 18:22:54.616306 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.610898    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.734342    2565 apiserver.go:52] "Watching apiserver"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.741354    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.741705    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-v227c","openshift-multus/multus-7zx75","openshift-network-diagnostics/network-check-target-mrffv","openshift-ovn-kubernetes/ovnkube-node-gx7xc","kube-system/konnectivity-agent-982jp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g","openshift-image-registry/node-ca-v67dg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal","openshift-multus/multus-additional-cni-plugins-csw9v","openshift-multus/network-metrics-daemon-6mmxv","openshift-network-operator/iptables-alerter-pkr6w","kube-system/global-pull-secret-syncer-tvpcl","kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6t8tb"]
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.743329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.744594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.745596    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.745766    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.745806    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.745942    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vx5b6\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.745978    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.746902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.746990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.747586    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.747622    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.747723    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.747788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k4wpl\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.748029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.748226    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.748268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qc96l\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.748262    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.748510    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.748585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.749449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.749748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750401    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750411    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750419    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750712    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750715    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750777    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.750836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-chfwp\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.751138    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.751191    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.751403    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64ktj\""
Jul 07 18:22:54.779241 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.751519    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.751903    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.752630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.752952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.753760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-glmcp\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.754175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.754259    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.754266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.754260    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.754324    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9sk2k\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.754714    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.755553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.756050    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.756092    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-79t47\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.757428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.757819    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.758002    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.758149    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.758292    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bhns6\""
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.759786    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.760990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.761087    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-log-socket\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsh6\" (UniqueName: \"kubernetes.io/projected/2b1cab1a-2f12-4056-b400-773d0b56b4a1-kube-api-access-cbsh6\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-var-lib-kubelet\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-cni-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.786763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-cnibin\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovnkube-script-lib\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2j7\" (UniqueName: \"kubernetes.io/projected/2b46e1ab-8205-43d7-a162-2bd5b6838f46-kube-api-access-tl2j7\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.770977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d8q2\" (UniqueName: \"kubernetes.io/projected/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-kube-api-access-7d8q2\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysconfig\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-systemd\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-socket-dir-parent\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1cab1a-2f12-4056-b400-773d0b56b4a1-tmp-dir\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-modprobe-d\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysctl-conf\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x687z\" (UniqueName: \"kubernetes.io/projected/4a81c30e-73a2-4430-ab86-5b970518906e-kube-api-access-x687z\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-cni-multus\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:54.787972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-env-overrides\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24w5\" (UniqueName: \"kubernetes.io/projected/deec1483-7b93-4fb6-8db4-904b9bac9a96-kube-api-access-w24w5\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-dbus\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f271ea9c-02de-4e46-8a14-16f67634a5c4-iptables-alerter-script\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-device-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cnibin\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-host\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-systemd\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkbv\" (UniqueName: \"kubernetes.io/projected/22c5160b-a1e1-404d-b045-bbbd68a66ef0-kube-api-access-gdkbv\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghfg\" (UniqueName: \"kubernetes.io/projected/f271ea9c-02de-4e46-8a14-16f67634a5c4-kube-api-access-kghfg\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbds5\" (UniqueName: \"kubernetes.io/projected/ac52d42f-712b-4920-bbd7-c52ca098facc-kube-api-access-lbds5\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.771940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-etc-selinux\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-kubelet\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysctl-d\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.788597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-socket-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-run-netns\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-etc-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5lg\" (UniqueName: \"kubernetes.io/projected/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-kube-api-access-sm5lg\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-run\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f271ea9c-02de-4e46-8a14-16f67634a5c4-host-slash\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c73f33e5-a952-480a-843a-b3cd9b8ae12a-agent-certs\") pod \"konnectivity-agent-982jp\" (UID: \"c73f33e5-a952-480a-843a-b3cd9b8ae12a\") " pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b46e1ab-8205-43d7-a162-2bd5b6838f46-serviceca\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4a81c30e-73a2-4430-ab86-5b970518906e-etc-tuned\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac52d42f-712b-4920-bbd7-c52ca098facc-cni-binary-copy\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-daemon-config\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-etc-kubernetes\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.789859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovnkube-config\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b1cab1a-2f12-4056-b400-773d0b56b4a1-hosts-file\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-systemd-units\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-ovn\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-slash\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b46e1ab-8205-43d7-a162-2bd5b6838f46-host\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-os-release\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-os-release\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-multus-certs\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-registration-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-kubelet-config\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-system-cni-dir\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-kubernetes\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.772993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-cni-bin\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-hostroot\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-conf-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.790423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-var-lib-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c73f33e5-a952-480a-843a-b3cd9b8ae12a-konnectivity-ca\") pod \"konnectivity-agent-982jp\" (UID: \"c73f33e5-a952-480a-843a-b3cd9b8ae12a\") " pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-lib-modules\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a81c30e-73a2-4430-ab86-5b970518906e-tmp\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-node-log\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-cni-netd\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-sys\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-system-cni-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-k8s-cni-cncf-io\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-kubelet\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-cni-bin\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-sys-fs\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.773396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-netns\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.791423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.774037    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 18:22:54.812925 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.812902    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ccpjz"
Jul 07 18:22:54.839137 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.839115    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ccpjz"
Jul 07 18:22:54.861827 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.861804    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 18:22:54.874510 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f271ea9c-02de-4e46-8a14-16f67634a5c4-iptables-alerter-script\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.874621 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-device-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.874712 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cnibin\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.874771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-host\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.874771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-systemd\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.874867 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkbv\" (UniqueName: \"kubernetes.io/projected/22c5160b-a1e1-404d-b045-bbbd68a66ef0-kube-api-access-gdkbv\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.874867 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.874867 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghfg\" (UniqueName: \"kubernetes.io/projected/f271ea9c-02de-4e46-8a14-16f67634a5c4-kube-api-access-kghfg\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.874867 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbds5\" (UniqueName: \"kubernetes.io/projected/ac52d42f-712b-4920-bbd7-c52ca098facc-kube-api-access-lbds5\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-etc-selinux\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-kubelet\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysctl-d\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-socket-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-run-netns\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.874976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-etc-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5lg\" (UniqueName: \"kubernetes.io/projected/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-kube-api-access-sm5lg\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-run\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.875083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f271ea9c-02de-4e46-8a14-16f67634a5c4-host-slash\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c73f33e5-a952-480a-843a-b3cd9b8ae12a-agent-certs\") pod \"konnectivity-agent-982jp\" (UID: \"c73f33e5-a952-480a-843a-b3cd9b8ae12a\") " pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875123    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f271ea9c-02de-4e46-8a14-16f67634a5c4-iptables-alerter-script\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b46e1ab-8205-43d7-a162-2bd5b6838f46-serviceca\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4a81c30e-73a2-4430-ab86-5b970518906e-etc-tuned\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac52d42f-712b-4920-bbd7-c52ca098facc-cni-binary-copy\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-daemon-config\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-etc-kubernetes\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875288    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysctl-d\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovnkube-config\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b1cab1a-2f12-4056-b400-773d0b56b4a1-hosts-file\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-systemd-units\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875349    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-device-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.875541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-ovn\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-slash\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b46e1ab-8205-43d7-a162-2bd5b6838f46-host\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875443    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-host\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-os-release\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875497    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-systemd\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-os-release\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875538    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-os-release\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-multus-certs\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-registration-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-kubelet-config\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-system-cni-dir\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-kubernetes\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875653    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-socket-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-cni-bin\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-hostroot\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875701    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-run-netns\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875749    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-hostroot\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-conf-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-var-lib-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875784    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-etc-kubernetes\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875795    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-etc-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c73f33e5-a952-480a-843a-b3cd9b8ae12a-konnectivity-ca\") pod \"konnectivity-agent-982jp\" (UID: \"c73f33e5-a952-480a-843a-b3cd9b8ae12a\") " pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-lib-modules\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a81c30e-73a2-4430-ab86-5b970518906e-tmp\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-node-log\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-cni-netd\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-sys\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-system-cni-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-k8s-cni-cncf-io\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876060    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-kubelet-config\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-kubelet\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-cni-bin\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-sys-fs\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.876988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-netns\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876222    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-log-socket\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsh6\" (UniqueName: \"kubernetes.io/projected/2b1cab1a-2f12-4056-b400-773d0b56b4a1-kube-api-access-cbsh6\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-var-lib-kubelet\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876289    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-daemon-config\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-cni-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-cnibin\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876343    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-run\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876363    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-conf-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.876385    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876435    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876405    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f271ea9c-02de-4e46-8a14-16f67634a5c4-host-slash\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.876488    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:55.376450873 +0000 UTC m=+2.952214762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.875407    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cnibin\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovnkube-script-lib\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876726    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-system-cni-dir\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.877726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876854    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-kubernetes\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.876907    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-cni-bin\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877168    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac52d42f-712b-4920-bbd7-c52ca098facc-cni-binary-copy\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877310    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b1cab1a-2f12-4056-b400-773d0b56b4a1-hosts-file\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877355    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-var-lib-openvswitch\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877391    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-systemd-units\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877484    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovnkube-config\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877776    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-lib-modules\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877822    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-run-ovn\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877845    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877943    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c73f33e5-a952-480a-843a-b3cd9b8ae12a-konnectivity-ca\") pod \"konnectivity-agent-982jp\" (UID: \"c73f33e5-a952-480a-843a-b3cd9b8ae12a\") " pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877948    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-slash\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.877994    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-node-log\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878028    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-cni-netd\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878066    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878098    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b46e1ab-8205-43d7-a162-2bd5b6838f46-host\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2j7\" (UniqueName: \"kubernetes.io/projected/2b46e1ab-8205-43d7-a162-2bd5b6838f46-kube-api-access-tl2j7\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878404    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-etc-selinux\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.878501 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d8q2\" (UniqueName: \"kubernetes.io/projected/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-kube-api-access-7d8q2\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysconfig\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-systemd\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-socket-dir-parent\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1cab1a-2f12-4056-b400-773d0b56b4a1-tmp-dir\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-modprobe-d\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysctl-conf\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878803    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysctl-conf\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878837    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-sysconfig\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878874    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-systemd\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.878904    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-socket-dir-parent\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.878953    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.878986    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:22:55.378974749 +0000 UTC m=+2.954738620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879022    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.879324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879088    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-kubelet\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879098    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879147    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-os-release\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879172    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1cab1a-2f12-4056-b400-773d0b56b4a1-tmp-dir\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x687z\" (UniqueName: \"kubernetes.io/projected/4a81c30e-73a2-4430-ab86-5b970518906e-kube-api-access-x687z\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-multus-certs\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-cni-multus\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879221    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-registration-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-env-overrides\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879290    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-sys\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24w5\" (UniqueName: \"kubernetes.io/projected/deec1483-7b93-4fb6-8db4-904b9bac9a96-kube-api-access-w24w5\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879331    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-kubelet\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-dbus\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879370    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-log-socket\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879409    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-cni-bin\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879508    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-dbus\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879613    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-system-cni-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879616    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-etc-modprobe-d\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879677    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-netns\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879716    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-cnibin\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879682    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a81c30e-73a2-4430-ab86-5b970518906e-var-lib-kubelet\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879761    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-sys-fs\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.879678    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-var-lib-cni-multus\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880110    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-multus-cni-dir\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880122    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b46e1ab-8205-43d7-a162-2bd5b6838f46-serviceca\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880133    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac52d42f-712b-4920-bbd7-c52ca098facc-host-run-k8s-cni-cncf-io\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880289    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovnkube-script-lib\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880399    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22c5160b-a1e1-404d-b045-bbbd68a66ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880493    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22c5160b-a1e1-404d-b045-bbbd68a66ef0-env-overrides\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880528    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/deec1483-7b93-4fb6-8db4-904b9bac9a96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.880988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.880824    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.881843 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.881821    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a81c30e-73a2-4430-ab86-5b970518906e-tmp\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.882515 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.882486    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4a81c30e-73a2-4430-ab86-5b970518906e-etc-tuned\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.882997 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.882976    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c73f33e5-a952-480a-843a-b3cd9b8ae12a-agent-certs\") pod \"konnectivity-agent-982jp\" (UID: \"c73f33e5-a952-480a-843a-b3cd9b8ae12a\") " pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:54.883446 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.883425    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22c5160b-a1e1-404d-b045-bbbd68a66ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.892007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.891985    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkbv\" (UniqueName: \"kubernetes.io/projected/22c5160b-a1e1-404d-b045-bbbd68a66ef0-kube-api-access-gdkbv\") pod \"ovnkube-node-gx7xc\" (UID: \"22c5160b-a1e1-404d-b045-bbbd68a66ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:54.894253 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.894213    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5lg\" (UniqueName: \"kubernetes.io/projected/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-kube-api-access-sm5lg\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:54.905010 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.904982    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:22:54.905010 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.905008    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:22:54.905157 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.905021    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:54.905157 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:54.905076    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:55.405059674 +0000 UTC m=+2.980823554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:54.907330 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.907275    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbds5\" (UniqueName: \"kubernetes.io/projected/ac52d42f-712b-4920-bbd7-c52ca098facc-kube-api-access-lbds5\") pod \"multus-7zx75\" (UID: \"ac52d42f-712b-4920-bbd7-c52ca098facc\") " pod="openshift-multus/multus-7zx75"
Jul 07 18:22:54.907570 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.907541    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsh6\" (UniqueName: \"kubernetes.io/projected/2b1cab1a-2f12-4056-b400-773d0b56b4a1-kube-api-access-cbsh6\") pod \"node-resolver-v227c\" (UID: \"2b1cab1a-2f12-4056-b400-773d0b56b4a1\") " pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:54.909111 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.909072    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d8q2\" (UniqueName: \"kubernetes.io/projected/c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e-kube-api-access-7d8q2\") pod \"multus-additional-cni-plugins-csw9v\" (UID: \"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e\") " pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:54.909306 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.909285    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2j7\" (UniqueName: \"kubernetes.io/projected/2b46e1ab-8205-43d7-a162-2bd5b6838f46-kube-api-access-tl2j7\") pod \"node-ca-v67dg\" (UID: \"2b46e1ab-8205-43d7-a162-2bd5b6838f46\") " pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:54.922606 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.922557    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x687z\" (UniqueName: \"kubernetes.io/projected/4a81c30e-73a2-4430-ab86-5b970518906e-kube-api-access-x687z\") pod \"tuned-6t8tb\" (UID: \"4a81c30e-73a2-4430-ab86-5b970518906e\") " pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:54.922606 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.922594    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24w5\" (UniqueName: \"kubernetes.io/projected/deec1483-7b93-4fb6-8db4-904b9bac9a96-kube-api-access-w24w5\") pod \"aws-ebs-csi-driver-node-56s6g\" (UID: \"deec1483-7b93-4fb6-8db4-904b9bac9a96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:54.923201 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:54.923184    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghfg\" (UniqueName: \"kubernetes.io/projected/f271ea9c-02de-4e46-8a14-16f67634a5c4-kube-api-access-kghfg\") pod \"iptables-alerter-pkr6w\" (UID: \"f271ea9c-02de-4e46-8a14-16f67634a5c4\") " pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:55.059868 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.059790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v67dg"
Jul 07 18:22:55.064982 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.064957    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-csw9v"
Jul 07 18:22:55.073539 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.073499    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46e1ab_8205_43d7_a162_2bd5b6838f46.slice/crio-105a068aa3f790bb39ab4e8306da5b99dcb4d723234cce8fe83b194c8bda047e WatchSource:0}: Error finding container 105a068aa3f790bb39ab4e8306da5b99dcb4d723234cce8fe83b194c8bda047e: Status 404 returned error can't find the container with id 105a068aa3f790bb39ab4e8306da5b99dcb4d723234cce8fe83b194c8bda047e
Jul 07 18:22:55.075432 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.075391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6t8tb"
Jul 07 18:22:55.080928 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.080901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:22:55.082759 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.082608    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c54eb5_f3ab_4cbe_9176_f19d17dbea7e.slice/crio-d26b40ba710b162772d54780142fb19d379c640ab17efdd5454808337b067d57 WatchSource:0}: Error finding container d26b40ba710b162772d54780142fb19d379c640ab17efdd5454808337b067d57: Status 404 returned error can't find the container with id d26b40ba710b162772d54780142fb19d379c640ab17efdd5454808337b067d57
Jul 07 18:22:55.088528 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.088472    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a81c30e_73a2_4430_ab86_5b970518906e.slice/crio-2fcf20a31d908fca4051ecf794620e230d86add62add5305845681f29464ae1e WatchSource:0}: Error finding container 2fcf20a31d908fca4051ecf794620e230d86add62add5305845681f29464ae1e: Status 404 returned error can't find the container with id 2fcf20a31d908fca4051ecf794620e230d86add62add5305845681f29464ae1e
Jul 07 18:22:55.090607 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.090518    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c5160b_a1e1_404d_b045_bbbd68a66ef0.slice/crio-a536b9fb3fcf95e02bef22b59388e35841b428d76a78b1f0162bed7ce8e2b0fc WatchSource:0}: Error finding container a536b9fb3fcf95e02bef22b59388e35841b428d76a78b1f0162bed7ce8e2b0fc: Status 404 returned error can't find the container with id a536b9fb3fcf95e02bef22b59388e35841b428d76a78b1f0162bed7ce8e2b0fc
Jul 07 18:22:55.092476 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.092323    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkr6w"
Jul 07 18:22:55.098489 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.098319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:22:55.101756 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.101733    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf271ea9c_02de_4e46_8a14_16f67634a5c4.slice/crio-417456377b347aee7d12fdbe9222bb6eb8b13b5a2e7a67bc99ee3481f42edcda WatchSource:0}: Error finding container 417456377b347aee7d12fdbe9222bb6eb8b13b5a2e7a67bc99ee3481f42edcda: Status 404 returned error can't find the container with id 417456377b347aee7d12fdbe9222bb6eb8b13b5a2e7a67bc99ee3481f42edcda
Jul 07 18:22:55.104205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.104064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g"
Jul 07 18:22:55.107719 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.107689    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73f33e5_a952_480a_843a_b3cd9b8ae12a.slice/crio-45c118455ba3f7674472e6ae1e30880df17340be57a1ee10f773cc930fd7103b WatchSource:0}: Error finding container 45c118455ba3f7674472e6ae1e30880df17340be57a1ee10f773cc930fd7103b: Status 404 returned error can't find the container with id 45c118455ba3f7674472e6ae1e30880df17340be57a1ee10f773cc930fd7103b
Jul 07 18:22:55.109242 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.109160    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7zx75"
Jul 07 18:22:55.114924 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.114902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v227c"
Jul 07 18:22:55.125354 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.125327    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1cab1a_2f12_4056_b400_773d0b56b4a1.slice/crio-294d69d796bc1913806eec2bd59091d0fa14c19755fc2d79ff82576648154873 WatchSource:0}: Error finding container 294d69d796bc1913806eec2bd59091d0fa14c19755fc2d79ff82576648154873: Status 404 returned error can't find the container with id 294d69d796bc1913806eec2bd59091d0fa14c19755fc2d79ff82576648154873
Jul 07 18:22:55.126044 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:22:55.126014    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac52d42f_712b_4920_bbd7_c52ca098facc.slice/crio-08ee6d1560054ea84a9ac8bd96199c972d0551e7c3137e09a0df7b8967e8041d WatchSource:0}: Error finding container 08ee6d1560054ea84a9ac8bd96199c972d0551e7c3137e09a0df7b8967e8041d: Status 404 returned error can't find the container with id 08ee6d1560054ea84a9ac8bd96199c972d0551e7c3137e09a0df7b8967e8041d
Jul 07 18:22:55.382830 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.382706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:55.382830 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.382816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:55.383033 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.382880    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:55.383033 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.382929    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:55.383033 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.382948    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:22:56.382927019 +0000 UTC m=+3.958690892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:55.383033 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.382976    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:56.382961435 +0000 UTC m=+3.958725313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:55.483784 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.483752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:55.483952 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.483900    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:22:55.483952 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.483918    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:22:55.483952 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.483930    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:55.484081 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:55.483979    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:56.483966158 +0000 UTC m=+4.059730029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:55.840810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.840679    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:17:54 +0000 UTC" deadline="2028-03-09 17:55:59.10093077 +0000 UTC"
Jul 07 18:22:55.840810 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.840713    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14663h33m3.260222529s"
Jul 07 18:22:55.922938 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:55.922646    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 18:22:56.042112 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.042080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:56.042305 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.042090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:56.042305 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.042208    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:22:56.042305 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.042223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:56.042305 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.042283    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:22:56.042611 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.042352    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:22:56.051751 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.051721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g" event={"ID":"deec1483-7b93-4fb6-8db4-904b9bac9a96","Type":"ContainerStarted","Data":"57f3b20ee3f87184a942a1aa13bedb2614dbeb0ff9e4551c21d9263971793212"}
Jul 07 18:22:56.053999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.053973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkr6w" event={"ID":"f271ea9c-02de-4e46-8a14-16f67634a5c4","Type":"ContainerStarted","Data":"417456377b347aee7d12fdbe9222bb6eb8b13b5a2e7a67bc99ee3481f42edcda"}
Jul 07 18:22:56.057711 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.057686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7zx75" event={"ID":"ac52d42f-712b-4920-bbd7-c52ca098facc","Type":"ContainerStarted","Data":"08ee6d1560054ea84a9ac8bd96199c972d0551e7c3137e09a0df7b8967e8041d"}
Jul 07 18:22:56.059416 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.059318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-982jp" event={"ID":"c73f33e5-a952-480a-843a-b3cd9b8ae12a","Type":"ContainerStarted","Data":"45c118455ba3f7674472e6ae1e30880df17340be57a1ee10f773cc930fd7103b"}
Jul 07 18:22:56.060919 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.060813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"a536b9fb3fcf95e02bef22b59388e35841b428d76a78b1f0162bed7ce8e2b0fc"}
Jul 07 18:22:56.062309 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.062236    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6t8tb" event={"ID":"4a81c30e-73a2-4430-ab86-5b970518906e","Type":"ContainerStarted","Data":"2fcf20a31d908fca4051ecf794620e230d86add62add5305845681f29464ae1e"}
Jul 07 18:22:56.064726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.064644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerStarted","Data":"d26b40ba710b162772d54780142fb19d379c640ab17efdd5454808337b067d57"}
Jul 07 18:22:56.067031 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.066964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v67dg" event={"ID":"2b46e1ab-8205-43d7-a162-2bd5b6838f46","Type":"ContainerStarted","Data":"105a068aa3f790bb39ab4e8306da5b99dcb4d723234cce8fe83b194c8bda047e"}
Jul 07 18:22:56.069893 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.069836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v227c" event={"ID":"2b1cab1a-2f12-4056-b400-773d0b56b4a1","Type":"ContainerStarted","Data":"294d69d796bc1913806eec2bd59091d0fa14c19755fc2d79ff82576648154873"}
Jul 07 18:22:56.391517 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.390808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:56.391517 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.390872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:56.391517 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.391007    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:56.391517 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.391066    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:58.391045479 +0000 UTC m=+5.966809358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:56.391517 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.391422    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:56.391517 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.391484    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:22:58.391449976 +0000 UTC m=+5.967213849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:56.491324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.491290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:56.491564 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.491497    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:22:56.491564 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.491520    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:22:56.491564 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.491533    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:56.491750 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:56.491587    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:58.491569517 +0000 UTC m=+6.067333393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:56.841780 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.841662    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:17:54 +0000 UTC" deadline="2028-01-22 19:41:25.472538102 +0000 UTC"
Jul 07 18:22:56.841780 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:56.841700    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13537h18m28.63084253s"
Jul 07 18:22:58.041743 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:58.041708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:58.042212 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.041832    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:22:58.042280 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:58.042213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:58.042333 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.042301    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:22:58.042385 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:58.042363    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:58.042479 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.042443    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:22:58.406201 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:58.405436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:22:58.406201 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:58.405532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:22:58.406201 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.405655    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:58.406201 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.405717    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:23:02.405697866 +0000 UTC m=+9.981461743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:22:58.406201 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.406079    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:58.406201 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.406125    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:02.406108977 +0000 UTC m=+9.981872856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:22:58.506713 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:22:58.506069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:22:58.506713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.506252    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:22:58.506713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.506274    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:22:58.506713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.506286    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:22:58.506713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:22:58.506343    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:02.506323386 +0000 UTC m=+10.082087262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:00.041707 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:00.041670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:00.042172 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:00.041713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:00.042172 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:00.041715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:00.042172 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:00.041810    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:00.042172 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:00.041901    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:00.042172 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:00.041991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:02.043197 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:02.042508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:02.043197 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.042636    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:02.043197 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:02.042995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:02.043197 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.043110    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:02.043197 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:02.043172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:02.043847 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.043238    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:02.434574 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:02.434490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:02.434726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:02.434656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:02.434726 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.434658    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:23:02.434834 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.434726    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:10.434707984 +0000 UTC m=+18.010471863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:23:02.434834 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.434768    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:23:02.434834 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.434812    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:23:10.43480046 +0000 UTC m=+18.010564332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:23:02.535225 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:02.535189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:02.535396 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.535340    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:23:02.535396 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.535359    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:23:02.535396 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.535372    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:02.535577 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:02.535436    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:10.535417772 +0000 UTC m=+18.111181657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:04.042092 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:04.042056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:04.042092 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:04.042088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:04.042629 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:04.042090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:04.042629 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:04.042199    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:04.042629 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:04.042252    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:04.042629 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:04.042325    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:06.042737 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.042487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:06.043209 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.042507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:06.043209 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:06.042853    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:06.043209 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.042507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:06.043209 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:06.042910    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:06.043209 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:06.042965    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:06.093608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.093362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6t8tb" event={"ID":"4a81c30e-73a2-4430-ab86-5b970518906e","Type":"ContainerStarted","Data":"fafcbf8e2e13ac6ee6ebb671e2d18f3544935f4d07afa595c0d87ce65ac370a2"}
Jul 07 18:23:06.097058 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.097028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal" event={"ID":"bd00b61865beafaa890243361850b718","Type":"ContainerStarted","Data":"3c2dd7b0693274ac0ec42177e13e310a578735cd6401ff4ed6573ed83c7e1c05"}
Jul 07 18:23:06.112829 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.112772    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6t8tb" podStartSLOduration=3.659699222 podStartE2EDuration="14.112760101s" totalImagesPullingTime="10.453060879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:06.111737372 +0000 UTC m=+13.687501270" watchObservedRunningTime="2026-07-07 18:23:06.112760101 +0000 UTC m=+13.688523994"
Jul 07 18:23:06.129760 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:06.129724    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-31.ec2.internal" podStartSLOduration=12.129706027 podStartE2EDuration="12.129706027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:06.129603425 +0000 UTC m=+13.705367319" watchObservedRunningTime="2026-07-07 18:23:06.129706027 +0000 UTC m=+13.705469924"
Jul 07 18:23:07.100487 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.100249    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e" containerID="b313fd849ce669d355a06fe6fa59b0a3c5bf249952118d6665c9f629a551e17c" exitCode=0
Jul 07 18:23:07.100487 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.100335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerDied","Data":"b313fd849ce669d355a06fe6fa59b0a3c5bf249952118d6665c9f629a551e17c"}
Jul 07 18:23:07.102177 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.102153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v67dg" event={"ID":"2b46e1ab-8205-43d7-a162-2bd5b6838f46","Type":"ContainerStarted","Data":"1065d37bd82b3d8cdf223f5ead9a33a530d2ca3eab7d2b78f95572d42ea2fcb0"}
Jul 07 18:23:07.103753 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.103721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v227c" event={"ID":"2b1cab1a-2f12-4056-b400-773d0b56b4a1","Type":"ContainerStarted","Data":"9ab683fbb1bd64489ba64a60646787b62097ae467bfde220986de6016ac5d0cc"}
Jul 07 18:23:07.105729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.105675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g" event={"ID":"deec1483-7b93-4fb6-8db4-904b9bac9a96","Type":"ContainerStarted","Data":"54ebbc2f06da165c17a206974920a39e57d80150734b3ae9bf57e943d6409c09"}
Jul 07 18:23:07.107343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.107319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkr6w" event={"ID":"f271ea9c-02de-4e46-8a14-16f67634a5c4","Type":"ContainerStarted","Data":"c73c7f2521db8eed9fa3daad5fe885aca316a3e15ff601a342c79d89128e632a"}
Jul 07 18:23:07.109009 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.108986    2565 generic.go:356] "Generic (PLEG): container finished" podID="05b4edd690f8cd78d388db58e1fdf50f" containerID="d74b282a3f8a894f3971598fec2d5fb8be3dab9a2a93e8f396d8f96dfdef3273" exitCode=0
Jul 07 18:23:07.109112 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.109073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal" event={"ID":"05b4edd690f8cd78d388db58e1fdf50f","Type":"ContainerDied","Data":"d74b282a3f8a894f3971598fec2d5fb8be3dab9a2a93e8f396d8f96dfdef3273"}
Jul 07 18:23:07.110505 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.110482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-982jp" event={"ID":"c73f33e5-a952-480a-843a-b3cd9b8ae12a","Type":"ContainerStarted","Data":"07fed9f9f3fa9b4de63e8e8b8edbe19e2464241f543413eb162e532e007bbf78"}
Jul 07 18:23:07.138327 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.138279    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pkr6w" podStartSLOduration=5.119622058 podStartE2EDuration="15.138263392s" totalImagesPullingTime="10.018641334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:07.138054952 +0000 UTC m=+14.713818901" watchObservedRunningTime="2026-07-07 18:23:07.138263392 +0000 UTC m=+14.714027287"
Jul 07 18:23:07.156259 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.156219    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-982jp" podStartSLOduration=4.092759157 podStartE2EDuration="14.156202791s" totalImagesPullingTime="10.063443634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:07.155694532 +0000 UTC m=+14.731458445" watchObservedRunningTime="2026-07-07 18:23:07.156202791 +0000 UTC m=+14.731966686"
Jul 07 18:23:07.170469 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.170427    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v227c" podStartSLOduration=4.126713616 podStartE2EDuration="14.170417278s" totalImagesPullingTime="10.043703662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:07.169997046 +0000 UTC m=+14.745760945" watchObservedRunningTime="2026-07-07 18:23:07.170417278 +0000 UTC m=+14.746181173"
Jul 07 18:23:07.203357 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:07.203315    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v67dg" podStartSLOduration=5.158260458 podStartE2EDuration="15.203299709s" totalImagesPullingTime="10.045039251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:07.203021357 +0000 UTC m=+14.778785253" watchObservedRunningTime="2026-07-07 18:23:07.203299709 +0000 UTC m=+14.779063604"
Jul 07 18:23:08.041708 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:08.041674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:08.041918 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:08.041826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:08.041918 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:08.041848    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:08.042045 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:08.041924    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:08.042045 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:08.041940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:08.042144 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:08.042041    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:08.111554 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:08.111522    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:23:08.112747 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:08.112721    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:23:09.114573 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:09.114547    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 18:23:10.041924 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:10.041891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:10.041924 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:10.041934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:10.042196 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:10.042015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:10.042196 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.042033    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:10.042196 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.042134    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:10.042317 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.042207    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:10.502897 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:10.502820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:10.503512 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:10.502915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:10.503512 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.502976    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:23:10.503512 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.503022    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:23:10.503512 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.503046    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:26.503027066 +0000 UTC m=+34.078790947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:23:10.503512 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.503064    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:23:26.503054827 +0000 UTC m=+34.078818704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:23:10.603666 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:10.603618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:10.603809 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.603791    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:23:10.603849 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.603813    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:23:10.603849 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.603822    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:10.603922 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:10.603872    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:26.603857678 +0000 UTC m=+34.179621555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:12.041581 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:12.041540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:12.042015 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:12.041550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:12.042015 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:12.041678    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:12.042015 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:12.041676    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:12.042015 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:12.041793    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:12.042015 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:12.041886    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:14.042358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:14.042324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:14.042895 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:14.042432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:14.042895 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:14.042444    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:14.042895 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:14.042481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:14.042895 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:14.042619    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:14.042895 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:14.042705    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:15.127341 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.126973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal" event={"ID":"05b4edd690f8cd78d388db58e1fdf50f","Type":"ContainerStarted","Data":"4ff8e314d0ed48a3bf7743d25db5570e6e79e0f447782f00fe06970088aec3d0"}
Jul 07 18:23:15.129195 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.129159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7zx75" event={"ID":"ac52d42f-712b-4920-bbd7-c52ca098facc","Type":"ContainerStarted","Data":"c9a8f716af7634af76a3c922f4b1018b89462c9d1961f174c8b5a476000bdf20"}
Jul 07 18:23:15.132031 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.132010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"03db2868557d61299c540f345e6f4358609c3b4249d8058507a25aee21adb14f"}
Jul 07 18:23:15.132124 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.132072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"7fc043996b048d98a98fcece845b4cef7842d12d9bc90e7ab482c8ff56477c88"}
Jul 07 18:23:15.132124 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.132111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"0e50f52ec07228a35809bddff668774302d81eb11b274ca97af2a9a8a935b7dc"}
Jul 07 18:23:15.132205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.132151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"4603f091093259fdbeeb7af2cb1e127eddbd970aee57d5a7d23d21f7ddf62e3c"}
Jul 07 18:23:15.143776 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.143718    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-31.ec2.internal" podStartSLOduration=21.143682501 podStartE2EDuration="21.143682501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:15.143533247 +0000 UTC m=+22.719297144" watchObservedRunningTime="2026-07-07 18:23:15.143682501 +0000 UTC m=+22.719446387"
Jul 07 18:23:15.163711 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.163655    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7zx75" podStartSLOduration=2.514653404 podStartE2EDuration="22.163624436s" totalImagesPullingTime="19.648971032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:15.163245617 +0000 UTC m=+22.739009512" watchObservedRunningTime="2026-07-07 18:23:15.163624436 +0000 UTC m=+22.739388330"
Jul 07 18:23:15.839223 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.839194    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 18:23:15.841233 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.841158    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T18:23:15.839212555Z","UUID":"96176044-2f5a-4da2-b758-7cc9512776ab","Handler":null,"Name":"","Endpoint":""}
Jul 07 18:23:15.842722 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.842701    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 18:23:15.842816 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:15.842737    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 18:23:16.042185 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:16.042108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:16.042185 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:16.042154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:16.042395 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:16.042109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:16.042395 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:16.042233    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:16.042395 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:16.042295    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:16.042608 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:16.042401    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:16.137153 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:16.137124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"3d808eaa388b5937ddc3ae68597a5709c73fbefe80d55698d7e2ccd93e6bd181"}
Jul 07 18:23:16.137563 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:16.137193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"a9f44402dfff32093c0dbcc88f42c9e80e24796bc127db39759083d9d6de1dc5"}
Jul 07 18:23:16.138826 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:16.138800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g" event={"ID":"deec1483-7b93-4fb6-8db4-904b9bac9a96","Type":"ContainerStarted","Data":"35bd4d5ef4fbc7eace547e5fbcf901d3a16d632b3f5478db0a3be2c01cc4e075"}
Jul 07 18:23:18.042203 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:18.042171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:18.042796 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:18.042168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:18.042796 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:18.042294    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:18.042796 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:18.042170    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:18.042796 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:18.042348    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:18.042796 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:18.042425    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:18.146862 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:18.146819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"8327082b26efeab5b2bcc749b2c4392e0eff1ce77ce34f497b5a4b990c276978"}
Jul 07 18:23:18.148751 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:18.148726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g" event={"ID":"deec1483-7b93-4fb6-8db4-904b9bac9a96","Type":"ContainerStarted","Data":"7022a08c5ff10dabce6f2289f1c036954c0ce2278e9af65b9a1cf9a8cbbb989b"}
Jul 07 18:23:18.170497 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:18.170429    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-56s6g" podStartSLOduration=4.261911199 podStartE2EDuration="25.17041695s" totalImagesPullingTime="20.908505751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:18.170283667 +0000 UTC m=+25.746047583" watchObservedRunningTime="2026-07-07 18:23:18.17041695 +0000 UTC m=+25.746180843"
Jul 07 18:23:19.455015 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:19.454976    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:23:19.455503 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:19.455125    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 18:23:19.455651 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:19.455621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-982jp"
Jul 07 18:23:20.041922 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:20.041851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:20.042073 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:20.041851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:20.042073 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:20.041992    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:20.042073 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:20.041851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:20.042073 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:20.042059    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:20.042274 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:20.042156    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:21.159383 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.159148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" event={"ID":"22c5160b-a1e1-404d-b045-bbbd68a66ef0","Type":"ContainerStarted","Data":"bf422eb6eba7d8ba4958d175edfd535ec85ee679d17ac42c97d36048d71c50c7"}
Jul 07 18:23:21.159952 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.159449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:23:21.159952 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.159523    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:23:21.161659 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.161628    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e" containerID="557200421e5f871689a57785abb94f1dfc7028b0cd32938e5633ed4618458335" exitCode=0
Jul 07 18:23:21.161789 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.161693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerDied","Data":"557200421e5f871689a57785abb94f1dfc7028b0cd32938e5633ed4618458335"}
Jul 07 18:23:21.172892 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.172868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:23:21.175057 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.175036    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:23:21.175141 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.175129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:23:21.190162 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:21.190126    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc" podStartSLOduration=9.495566387 podStartE2EDuration="29.190112977s" totalImagesPullingTime="19.69454659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:21.189726952 +0000 UTC m=+28.765490865" watchObservedRunningTime="2026-07-07 18:23:21.190112977 +0000 UTC m=+28.765876861"
Jul 07 18:23:22.042114 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:22.042085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:22.042282 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:22.042085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:22.042282 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:22.042212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:22.042282 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:22.042085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:22.042436 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:22.042315    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:22.042436 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:22.042249    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:23.049432 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.049400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrffv"]
Jul 07 18:23:23.049911 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.049520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:23.049911 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:23.049635    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:23.050202 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.050181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6mmxv"]
Jul 07 18:23:23.050328 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.050267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:23.050388 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:23.050363    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:23.050784 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.050764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvpcl"]
Jul 07 18:23:23.050858 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.050845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:23.050969 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:23.050946    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:23.167889 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.167837    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e" containerID="a1b2a8e9a5e1a06381b87836fc7ca49037e9164f84367b989c7e20944f46a137" exitCode=0
Jul 07 18:23:23.167974 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:23.167922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerDied","Data":"a1b2a8e9a5e1a06381b87836fc7ca49037e9164f84367b989c7e20944f46a137"}
Jul 07 18:23:25.041938 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:25.041759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:25.042596 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:25.041759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:25.042596 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:25.042028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:25.042596 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:25.041759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:25.042596 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:25.042084    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:25.042596 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:25.042174    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:25.175474 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:25.175423    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e" containerID="bb7da2084e1301bfa21628c3c235982e6e689717b6ce01d19c29efc51c66366d" exitCode=0
Jul 07 18:23:25.175608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:25.175484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerDied","Data":"bb7da2084e1301bfa21628c3c235982e6e689717b6ce01d19c29efc51c66366d"}
Jul 07 18:23:26.531306 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:26.531267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:26.531306 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:26.531314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:26.531911 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.531428    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:23:26.531911 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.531491    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 18:23:26.531911 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.531509    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs podName:b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c nodeName:}" failed. No retries permitted until 2026-07-07 18:23:58.531492764 +0000 UTC m=+66.107256640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs") pod "network-metrics-daemon-6mmxv" (UID: "b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 18:23:26.531911 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.531532    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret podName:b7e28baa-d67e-4cb0-b1f4-7f38600ea365 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:58.531520659 +0000 UTC m=+66.107284535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret") pod "global-pull-secret-syncer-tvpcl" (UID: "b7e28baa-d67e-4cb0-b1f4-7f38600ea365") : object "kube-system"/"original-pull-secret" not registered
Jul 07 18:23:26.632126 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:26.632093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:26.632266 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.632233    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 18:23:26.632266 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.632247    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 18:23:26.632266 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.632256    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5gwxq for pod openshift-network-diagnostics/network-check-target-mrffv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:26.632364 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:26.632297    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq podName:7b130aef-ae7c-4338-bf24-234e14252958 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:58.632285387 +0000 UTC m=+66.208049262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5gwxq" (UniqueName: "kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq") pod "network-check-target-mrffv" (UID: "7b130aef-ae7c-4338-bf24-234e14252958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 18:23:27.041855 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.041821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:27.041855 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.041836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:27.042126 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:27.041961    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mmxv" podUID="b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c"
Jul 07 18:23:27.042126 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:27.042028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvpcl" podUID="b7e28baa-d67e-4cb0-b1f4-7f38600ea365"
Jul 07 18:23:27.042126 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.042083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:27.042284 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:27.042171    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrffv" podUID="7b130aef-ae7c-4338-bf24-234e14252958"
Jul 07 18:23:27.722045 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.722015    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-31.ec2.internal" event="NodeReady"
Jul 07 18:23:27.722513 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.722181    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 18:23:27.766718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.766689    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d9f4cdd64-m72g2"]
Jul 07 18:23:27.768507 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.768491    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.773619 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.773593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 18:23:27.777393 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.777373    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-944sl\""
Jul 07 18:23:27.777638 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.777621    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 18:23:27.777691 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.777626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 18:23:27.779311 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.779296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 18:23:27.783396 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.781624    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fbx64"]
Jul 07 18:23:27.784186 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.784162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:27.784999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.784981    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rg6xl"]
Jul 07 18:23:27.786689 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.786671    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:27.786795 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.786723    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 18:23:27.787125 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.787109    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 18:23:27.787204 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.787116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9fz64\""
Jul 07 18:23:27.791597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.791579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 18:23:27.791703 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.791668    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 18:23:27.792019 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.792004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 18:23:27.794740 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.794724    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xk8xw\""
Jul 07 18:23:27.795024 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.795005    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9f4cdd64-m72g2"]
Jul 07 18:23:27.802051 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.802031    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fbx64"]
Jul 07 18:23:27.810069 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.810044    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rg6xl"]
Jul 07 18:23:27.941983 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.941907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ls8\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-kube-api-access-s8ls8\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.941983 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.941945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpnw\" (UniqueName: \"kubernetes.io/projected/4bce18d9-e01e-42d4-83c7-c40dfc7fc121-kube-api-access-wvpnw\") pod \"ingress-canary-rg6xl\" (UID: \"4bce18d9-e01e-42d4-83c7-c40dfc7fc121\") " pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.941984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bce18d9-e01e-42d4-83c7-c40dfc7fc121-cert\") pod \"ingress-canary-rg6xl\" (UID: \"4bce18d9-e01e-42d4-83c7-c40dfc7fc121\") " pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-trusted-ca\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-bound-sa-token\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93285c2-1bc0-4c94-8490-207d1ed3072a-ca-trust-extracted\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43eb2148-3f6d-4407-a55a-a63d49dc7324-config-volume\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:27.942163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eb2148-3f6d-4407-a55a-a63d49dc7324-tmp-dir\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:27.942425 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-image-registry-private-configuration\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.942425 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-certificates\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.942425 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gznd\" (UniqueName: \"kubernetes.io/projected/43eb2148-3f6d-4407-a55a-a63d49dc7324-kube-api-access-5gznd\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:27.942425 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:27.942425 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:27.942247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-installation-pull-secrets\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043328 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gznd\" (UniqueName: \"kubernetes.io/projected/43eb2148-3f6d-4407-a55a-a63d49dc7324-kube-api-access-5gznd\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.043328 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-installation-pull-secrets\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ls8\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-kube-api-access-s8ls8\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpnw\" (UniqueName: \"kubernetes.io/projected/4bce18d9-e01e-42d4-83c7-c40dfc7fc121-kube-api-access-wvpnw\") pod \"ingress-canary-rg6xl\" (UID: \"4bce18d9-e01e-42d4-83c7-c40dfc7fc121\") " pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bce18d9-e01e-42d4-83c7-c40dfc7fc121-cert\") pod \"ingress-canary-rg6xl\" (UID: \"4bce18d9-e01e-42d4-83c7-c40dfc7fc121\") " pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-trusted-ca\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.043490    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-bound-sa-token\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.043511    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f4cdd64-m72g2: secret "image-registry-tls" not found
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93285c2-1bc0-4c94-8490-207d1ed3072a-ca-trust-extracted\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.043565    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls podName:b93285c2-1bc0-4c94-8490-207d1ed3072a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:28.543544917 +0000 UTC m=+36.119308815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls") pod "image-registry-7d9f4cdd64-m72g2" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a") : secret "image-registry-tls" not found
Jul 07 18:23:28.043595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.044105 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43eb2148-3f6d-4407-a55a-a63d49dc7324-config-volume\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.044105 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eb2148-3f6d-4407-a55a-a63d49dc7324-tmp-dir\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.044105 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-image-registry-private-configuration\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.044105 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-certificates\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.044105 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.043982    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93285c2-1bc0-4c94-8490-207d1ed3072a-ca-trust-extracted\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.044105 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.044077    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 18:23:28.044397 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.044122    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls podName:43eb2148-3f6d-4407-a55a-a63d49dc7324 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:28.544106774 +0000 UTC m=+36.119870646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls") pod "dns-default-fbx64" (UID: "43eb2148-3f6d-4407-a55a-a63d49dc7324") : secret "dns-default-metrics-tls" not found
Jul 07 18:23:28.044397 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.044365    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eb2148-3f6d-4407-a55a-a63d49dc7324-tmp-dir\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.044523 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.044428    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-certificates\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.044523 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.044439    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43eb2148-3f6d-4407-a55a-a63d49dc7324-config-volume\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.044674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.044654    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-trusted-ca\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.048022 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.048001    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-installation-pull-secrets\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.048114 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.048027    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-image-registry-private-configuration\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.048114 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.048006    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bce18d9-e01e-42d4-83c7-c40dfc7fc121-cert\") pod \"ingress-canary-rg6xl\" (UID: \"4bce18d9-e01e-42d4-83c7-c40dfc7fc121\") " pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:28.053767 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.053731    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpnw\" (UniqueName: \"kubernetes.io/projected/4bce18d9-e01e-42d4-83c7-c40dfc7fc121-kube-api-access-wvpnw\") pod \"ingress-canary-rg6xl\" (UID: \"4bce18d9-e01e-42d4-83c7-c40dfc7fc121\") " pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:28.054533 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.054510    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-bound-sa-token\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.054894 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.054868    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ls8\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-kube-api-access-s8ls8\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.055240 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.055223    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gznd\" (UniqueName: \"kubernetes.io/projected/43eb2148-3f6d-4407-a55a-a63d49dc7324-kube-api-access-5gznd\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.099477 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.099435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rg6xl"
Jul 07 18:23:28.256901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.256705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rg6xl"]
Jul 07 18:23:28.262089 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:28.262061    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bce18d9_e01e_42d4_83c7_c40dfc7fc121.slice/crio-b4d4a3da0e76d083cdaa44c23d79f4eb0570f8d0722e042275a4e1b1f43c74db WatchSource:0}: Error finding container b4d4a3da0e76d083cdaa44c23d79f4eb0570f8d0722e042275a4e1b1f43c74db: Status 404 returned error can't find the container with id b4d4a3da0e76d083cdaa44c23d79f4eb0570f8d0722e042275a4e1b1f43c74db
Jul 07 18:23:28.546690 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.546571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:28.546690 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:28.546640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:28.546894 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.546753    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 18:23:28.546894 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.546785    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f4cdd64-m72g2: secret "image-registry-tls" not found
Jul 07 18:23:28.546894 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.546757    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 18:23:28.547009 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.546860    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls podName:b93285c2-1bc0-4c94-8490-207d1ed3072a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:29.546838593 +0000 UTC m=+37.122602476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls") pod "image-registry-7d9f4cdd64-m72g2" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a") : secret "image-registry-tls" not found
Jul 07 18:23:28.547009 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:28.546921    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls podName:43eb2148-3f6d-4407-a55a-a63d49dc7324 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:29.546907332 +0000 UTC m=+37.122671203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls") pod "dns-default-fbx64" (UID: "43eb2148-3f6d-4407-a55a-a63d49dc7324") : secret "dns-default-metrics-tls" not found
Jul 07 18:23:29.042066 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.041984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:29.042651 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.041983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:29.042651 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.041984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:29.046755 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.046735    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 18:23:29.046922 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.046781    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dxsjk\""
Jul 07 18:23:29.047778 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.047568    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 18:23:29.047778 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.047602    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wxf5l\""
Jul 07 18:23:29.047778 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.047692    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 18:23:29.047778 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.047704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 18:23:29.187039 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.187002    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rg6xl" event={"ID":"4bce18d9-e01e-42d4-83c7-c40dfc7fc121","Type":"ContainerStarted","Data":"b4d4a3da0e76d083cdaa44c23d79f4eb0570f8d0722e042275a4e1b1f43c74db"}
Jul 07 18:23:29.555966 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.555936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:29.556148 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.555990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:29.556148 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:29.556096    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 18:23:29.556261 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:29.556151    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls podName:43eb2148-3f6d-4407-a55a-a63d49dc7324 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:31.556133518 +0000 UTC m=+39.131897395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls") pod "dns-default-fbx64" (UID: "43eb2148-3f6d-4407-a55a-a63d49dc7324") : secret "dns-default-metrics-tls" not found
Jul 07 18:23:29.556261 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:29.556097    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 18:23:29.556261 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:29.556197    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f4cdd64-m72g2: secret "image-registry-tls" not found
Jul 07 18:23:29.556261 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:29.556260    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls podName:b93285c2-1bc0-4c94-8490-207d1ed3072a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:31.556245658 +0000 UTC m=+39.132009530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls") pod "image-registry-7d9f4cdd64-m72g2" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a") : secret "image-registry-tls" not found
Jul 07 18:23:29.964073 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.964047    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b885r"]
Jul 07 18:23:29.980110 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.980080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:29.983751 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.983595    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 18:23:29.983751 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.983626    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 18:23:29.983751 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.983641    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 18:23:29.983989 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.983753    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-62q56\""
Jul 07 18:23:29.984712 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.984696    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 18:23:29.995807 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:29.995779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b885r"]
Jul 07 18:23:30.162564 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.162479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1faf637a-833d-4e33-8902-66cee03aa98a-crio-socket\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.162564 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.162519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1faf637a-833d-4e33-8902-66cee03aa98a-data-volume\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.162996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.162608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrfl\" (UniqueName: \"kubernetes.io/projected/1faf637a-833d-4e33-8902-66cee03aa98a-kube-api-access-nwrfl\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.162996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.162653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.162996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.162684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.191978 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.191947    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e" containerID="e51beed77509b420b45fad2fa83d7488295ec599b86fb66d8f87f494654a3f0f" exitCode=0
Jul 07 18:23:30.192104 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.191982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerDied","Data":"e51beed77509b420b45fad2fa83d7488295ec599b86fb66d8f87f494654a3f0f"}
Jul 07 18:23:30.263355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.263330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.263577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.263425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1faf637a-833d-4e33-8902-66cee03aa98a-crio-socket\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.263577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.263453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1faf637a-833d-4e33-8902-66cee03aa98a-data-volume\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.263577 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.263484    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 18:23:30.263577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.263526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrfl\" (UniqueName: \"kubernetes.io/projected/1faf637a-833d-4e33-8902-66cee03aa98a-kube-api-access-nwrfl\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.263577 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.263571    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:30.763549675 +0000 UTC m=+38.339313565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : configmap "kube-rbac-proxy" not found
Jul 07 18:23:30.263904 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.263597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.263904 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.263697    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1faf637a-833d-4e33-8902-66cee03aa98a-crio-socket\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.263904 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.263719    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:30.263904 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.263760    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:30.763747077 +0000 UTC m=+38.339510954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:30.264114 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.264065    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1faf637a-833d-4e33-8902-66cee03aa98a-data-volume\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.275925 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.275901    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrfl\" (UniqueName: \"kubernetes.io/projected/1faf637a-833d-4e33-8902-66cee03aa98a-kube-api-access-nwrfl\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.653083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.653057    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"]
Jul 07 18:23:30.665102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.665084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"
Jul 07 18:23:30.667476 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.667438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqdx\" (UniqueName: \"kubernetes.io/projected/18f9a32c-8f0a-42fc-9413-c8eb30e28899-kube-api-access-8wqdx\") pod \"migrator-d57d58cc9-rfttw\" (UID: \"18f9a32c-8f0a-42fc-9413-c8eb30e28899\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"
Jul 07 18:23:30.668533 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.668514    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 18:23:30.669608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.669587    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7k2xn\""
Jul 07 18:23:30.669701 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.669587    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 18:23:30.671245 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.670856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"]
Jul 07 18:23:30.768284 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.768242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.768482 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.768362    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:30.768482 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.768379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqdx\" (UniqueName: \"kubernetes.io/projected/18f9a32c-8f0a-42fc-9413-c8eb30e28899-kube-api-access-8wqdx\") pod \"migrator-d57d58cc9-rfttw\" (UID: \"18f9a32c-8f0a-42fc-9413-c8eb30e28899\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"
Jul 07 18:23:30.768482 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.768430    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:31.768409553 +0000 UTC m=+39.344173429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:30.768664 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.768489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:30.768664 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.768579    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 18:23:30.768664 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:30.768650    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:31.768632787 +0000 UTC m=+39.344396676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : configmap "kube-rbac-proxy" not found
Jul 07 18:23:30.780485 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.780439    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqdx\" (UniqueName: \"kubernetes.io/projected/18f9a32c-8f0a-42fc-9413-c8eb30e28899-kube-api-access-8wqdx\") pod \"migrator-d57d58cc9-rfttw\" (UID: \"18f9a32c-8f0a-42fc-9413-c8eb30e28899\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"
Jul 07 18:23:30.974622 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:30.974550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"
Jul 07 18:23:31.137388 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.137354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw"]
Jul 07 18:23:31.141022 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:31.140988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f9a32c_8f0a_42fc_9413_c8eb30e28899.slice/crio-aed65ab11861833bab7fb7fc168493206a40c1bbb980c5c0e38906c5f4826f2e WatchSource:0}: Error finding container aed65ab11861833bab7fb7fc168493206a40c1bbb980c5c0e38906c5f4826f2e: Status 404 returned error can't find the container with id aed65ab11861833bab7fb7fc168493206a40c1bbb980c5c0e38906c5f4826f2e
Jul 07 18:23:31.197912 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.197875    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e" containerID="39051f4e1aab6b8f7264e908f6c87bb32151dd102b2e8443f65f81c0360246e6" exitCode=0
Jul 07 18:23:31.198312 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.197957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerDied","Data":"39051f4e1aab6b8f7264e908f6c87bb32151dd102b2e8443f65f81c0360246e6"}
Jul 07 18:23:31.199428 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.199403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw" event={"ID":"18f9a32c-8f0a-42fc-9413-c8eb30e28899","Type":"ContainerStarted","Data":"aed65ab11861833bab7fb7fc168493206a40c1bbb980c5c0e38906c5f4826f2e"}
Jul 07 18:23:31.576521 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.576326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:31.576713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.576506    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 18:23:31.576713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.576600    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f4cdd64-m72g2: secret "image-registry-tls" not found
Jul 07 18:23:31.576713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.576666    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls podName:b93285c2-1bc0-4c94-8490-207d1ed3072a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:35.576644809 +0000 UTC m=+43.152408681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls") pod "image-registry-7d9f4cdd64-m72g2" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a") : secret "image-registry-tls" not found
Jul 07 18:23:31.576713 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.576669    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 18:23:31.576877 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.576718    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls podName:43eb2148-3f6d-4407-a55a-a63d49dc7324 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:35.576703634 +0000 UTC m=+43.152467507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls") pod "dns-default-fbx64" (UID: "43eb2148-3f6d-4407-a55a-a63d49dc7324") : secret "dns-default-metrics-tls" not found
Jul 07 18:23:31.576877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.576585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:31.752377 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.752351    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v227c_2b1cab1a-2f12-4056-b400-773d0b56b4a1/dns-node-resolver/0.log"
Jul 07 18:23:31.778160 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.778126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:31.778303 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:31.778182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:31.778303 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.778255    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:31.778303 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.778280    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 18:23:31.778449 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.778333    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:33.778313067 +0000 UTC m=+41.354076946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:31.778449 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:31.778356    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:33.778342744 +0000 UTC m=+41.354106624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : configmap "kube-rbac-proxy" not found
Jul 07 18:23:32.206286 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.206251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csw9v" event={"ID":"c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e","Type":"ContainerStarted","Data":"096872f468f5d60e51891ba74f7fa7e696d184c3d17d904bfb84fcd18960a5ed"}
Jul 07 18:23:32.566605 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.566533    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v67dg_2b46e1ab-8205-43d7-a162-2bd5b6838f46/node-ca/0.log"
Jul 07 18:23:32.679865 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.679804    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-csw9v" podStartSLOduration=10.260259364 podStartE2EDuration="40.679779878s" totalImagesPullingTime="30.419520514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:32.264170279 +0000 UTC m=+39.839934174" watchObservedRunningTime="2026-07-07 18:23:32.679779878 +0000 UTC m=+40.255543771"
Jul 07 18:23:32.681165 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.681137    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-456pj"]
Jul 07 18:23:32.697769 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.697734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-456pj"]
Jul 07 18:23:32.697912 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.697847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.701363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.701344    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 18:23:32.701522 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.701394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 18:23:32.702098 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.702075    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 18:23:32.702202 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.702103    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-47px8\""
Jul 07 18:23:32.702202 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.702134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 18:23:32.785756 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.785729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f71f88f6-08cb-4a16-8d25-51f1f178fd10-signing-cabundle\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.785756 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.785759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f71f88f6-08cb-4a16-8d25-51f1f178fd10-signing-key\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.785954 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.785797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgf9p\" (UniqueName: \"kubernetes.io/projected/f71f88f6-08cb-4a16-8d25-51f1f178fd10-kube-api-access-xgf9p\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.886109 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.886085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f71f88f6-08cb-4a16-8d25-51f1f178fd10-signing-cabundle\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.886262 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.886115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f71f88f6-08cb-4a16-8d25-51f1f178fd10-signing-key\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.886262 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.886151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgf9p\" (UniqueName: \"kubernetes.io/projected/f71f88f6-08cb-4a16-8d25-51f1f178fd10-kube-api-access-xgf9p\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.886727 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.886705    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f71f88f6-08cb-4a16-8d25-51f1f178fd10-signing-cabundle\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.889390 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.889364    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f71f88f6-08cb-4a16-8d25-51f1f178fd10-signing-key\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:32.894577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:32.894558    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgf9p\" (UniqueName: \"kubernetes.io/projected/f71f88f6-08cb-4a16-8d25-51f1f178fd10-kube-api-access-xgf9p\") pod \"service-ca-758f599c5-456pj\" (UID: \"f71f88f6-08cb-4a16-8d25-51f1f178fd10\") " pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:33.008136 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:33.008109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-456pj"
Jul 07 18:23:33.308713 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:33.308550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-456pj"]
Jul 07 18:23:33.311741 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:33.311713    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71f88f6_08cb_4a16_8d25_51f1f178fd10.slice/crio-b9930a1885ea96e1e1336bffbc5f8b5118d5ecd0f5f0bc48eaebbe32ad1767f0 WatchSource:0}: Error finding container b9930a1885ea96e1e1336bffbc5f8b5118d5ecd0f5f0bc48eaebbe32ad1767f0: Status 404 returned error can't find the container with id b9930a1885ea96e1e1336bffbc5f8b5118d5ecd0f5f0bc48eaebbe32ad1767f0
Jul 07 18:23:33.794184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:33.794111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:33.794314 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:33.794198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:33.794314 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:33.794225    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 18:23:33.794314 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:33.794276    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:37.794261874 +0000 UTC m=+45.370025746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : configmap "kube-rbac-proxy" not found
Jul 07 18:23:33.794314 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:33.794284    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:33.794442 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:33.794323    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:37.79431177 +0000 UTC m=+45.370075643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:34.214703 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:34.214663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rg6xl" event={"ID":"4bce18d9-e01e-42d4-83c7-c40dfc7fc121","Type":"ContainerStarted","Data":"e958c93324aaec9c2bc165e4a1a3402d252be2755650662ff422287f1d98a0e1"}
Jul 07 18:23:34.217085 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:34.217054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw" event={"ID":"18f9a32c-8f0a-42fc-9413-c8eb30e28899","Type":"ContainerStarted","Data":"ec098757ffbe82811979197c3258c5fad0fd729200a174657830e1e5b3d4c75f"}
Jul 07 18:23:34.217294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:34.217271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw" event={"ID":"18f9a32c-8f0a-42fc-9413-c8eb30e28899","Type":"ContainerStarted","Data":"6b26e7dba87bb96156a359b4e7ce546128e070fbd84285ace416559815b0b16f"}
Jul 07 18:23:34.218285 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:34.218263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-456pj" event={"ID":"f71f88f6-08cb-4a16-8d25-51f1f178fd10","Type":"ContainerStarted","Data":"b9930a1885ea96e1e1336bffbc5f8b5118d5ecd0f5f0bc48eaebbe32ad1767f0"}
Jul 07 18:23:34.264651 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:34.264596    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rg6xl" podStartSLOduration=2.3538775259999998 podStartE2EDuration="7.264576723s" totalImagesPullingTime="4.910699197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:34.240034376 +0000 UTC m=+41.815798280" watchObservedRunningTime="2026-07-07 18:23:34.264576723 +0000 UTC m=+41.840340618"
Jul 07 18:23:34.264769 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:34.264702    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rfttw" podStartSLOduration=2.231143403 podStartE2EDuration="4.264695784s" totalImagesPullingTime="2.033552381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:34.263414614 +0000 UTC m=+41.839178511" watchObservedRunningTime="2026-07-07 18:23:34.264695784 +0000 UTC m=+41.840459683"
Jul 07 18:23:35.609162 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:35.609129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:35.609676 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:35.609249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:35.609676 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:35.609381    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 18:23:35.609676 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:35.609394    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9f4cdd64-m72g2: secret "image-registry-tls" not found
Jul 07 18:23:35.609676 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:35.609407    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 18:23:35.609676 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:35.609473    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls podName:b93285c2-1bc0-4c94-8490-207d1ed3072a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:43.609437643 +0000 UTC m=+51.185201516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls") pod "image-registry-7d9f4cdd64-m72g2" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a") : secret "image-registry-tls" not found
Jul 07 18:23:35.609676 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:35.609493    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls podName:43eb2148-3f6d-4407-a55a-a63d49dc7324 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:43.609483081 +0000 UTC m=+51.185246953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls") pod "dns-default-fbx64" (UID: "43eb2148-3f6d-4407-a55a-a63d49dc7324") : secret "dns-default-metrics-tls" not found
Jul 07 18:23:36.224267 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:36.224243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-456pj" event={"ID":"f71f88f6-08cb-4a16-8d25-51f1f178fd10","Type":"ContainerStarted","Data":"41988d0782d7a493bdf58b1f2db0aaa2f5d125fe3d1f2e0b6bd7545111a21021"}
Jul 07 18:23:37.827721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:37.827685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:37.828232 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:37.828206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:37.828490 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:37.828470    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 18:23:37.828651 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:37.828641    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:45.828620383 +0000 UTC m=+53.404384281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : configmap "kube-rbac-proxy" not found
Jul 07 18:23:37.829137 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:37.829115    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:37.829236 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:37.829163    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:23:45.829152008 +0000 UTC m=+53.404915880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:43.676850 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.676816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:43.677285 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.676884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:43.680795 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.680771    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43eb2148-3f6d-4407-a55a-a63d49dc7324-metrics-tls\") pod \"dns-default-fbx64\" (UID: \"43eb2148-3f6d-4407-a55a-a63d49dc7324\") " pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:43.680876 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.680805    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"image-registry-7d9f4cdd64-m72g2\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") " pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:43.693629 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.693609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:43.838217 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.838163    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-456pj" podStartSLOduration=9.033193397 podStartE2EDuration="11.838145688s" totalImagesPullingTime="2.804952291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:37.268609576 +0000 UTC m=+44.844373488" watchObservedRunningTime="2026-07-07 18:23:43.838145688 +0000 UTC m=+51.413909584"
Jul 07 18:23:43.838374 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.838287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fbx64"]
Jul 07 18:23:43.841320 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:43.841289    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43eb2148_3f6d_4407_a55a_a63d49dc7324.slice/crio-7a6fdf793c849b3e80434cf9891b858b95b711deeaab9eddfcae1f2d31febf73 WatchSource:0}: Error finding container 7a6fdf793c849b3e80434cf9891b858b95b711deeaab9eddfcae1f2d31febf73: Status 404 returned error can't find the container with id 7a6fdf793c849b3e80434cf9891b858b95b711deeaab9eddfcae1f2d31febf73
Jul 07 18:23:43.977894 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:43.977835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:44.129122 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:44.129092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9f4cdd64-m72g2"]
Jul 07 18:23:44.133435 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:44.133412    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93285c2_1bc0_4c94_8490_207d1ed3072a.slice/crio-80b069410d65ad8233a9b6faccd7e3b525ea82e16731ca7d099c470898fe1be7 WatchSource:0}: Error finding container 80b069410d65ad8233a9b6faccd7e3b525ea82e16731ca7d099c470898fe1be7: Status 404 returned error can't find the container with id 80b069410d65ad8233a9b6faccd7e3b525ea82e16731ca7d099c470898fe1be7
Jul 07 18:23:44.247684 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:44.247653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fbx64" event={"ID":"43eb2148-3f6d-4407-a55a-a63d49dc7324","Type":"ContainerStarted","Data":"7a6fdf793c849b3e80434cf9891b858b95b711deeaab9eddfcae1f2d31febf73"}
Jul 07 18:23:44.248974 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:44.248951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2" event={"ID":"b93285c2-1bc0-4c94-8490-207d1ed3072a","Type":"ContainerStarted","Data":"038246bd2ea90ca7154253be0181c3b7633fdcd44f121e430b074b8d212b5b1f"}
Jul 07 18:23:44.249064 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:44.248993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2" event={"ID":"b93285c2-1bc0-4c94-8490-207d1ed3072a","Type":"ContainerStarted","Data":"80b069410d65ad8233a9b6faccd7e3b525ea82e16731ca7d099c470898fe1be7"}
Jul 07 18:23:44.249064 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:44.249021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:23:44.278982 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:44.278929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2" podStartSLOduration=43.278911249 podStartE2EDuration="43.278911249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:44.278333841 +0000 UTC m=+51.854097738" watchObservedRunningTime="2026-07-07 18:23:44.278911249 +0000 UTC m=+51.854675144"
Jul 07 18:23:45.894940 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:45.894902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:45.894940 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:45.894946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:45.895412 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:45.895039    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:45.895412 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:45.895105    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls podName:1faf637a-833d-4e33-8902-66cee03aa98a nodeName:}" failed. No retries permitted until 2026-07-07 18:24:01.895089168 +0000 UTC m=+69.470853039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b885r" (UID: "1faf637a-833d-4e33-8902-66cee03aa98a") : secret "insights-runtime-extractor-tls" not found
Jul 07 18:23:45.895509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:45.895416    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1faf637a-833d-4e33-8902-66cee03aa98a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:23:47.259976 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:47.259936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fbx64" event={"ID":"43eb2148-3f6d-4407-a55a-a63d49dc7324","Type":"ContainerStarted","Data":"29ad7b6d31f08bd00e036bf5f8795eea255f50de720c098a9372f01ac4cfcad8"}
Jul 07 18:23:47.260338 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:47.259999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fbx64" event={"ID":"43eb2148-3f6d-4407-a55a-a63d49dc7324","Type":"ContainerStarted","Data":"743729e1639b8f9d2edc0b547824869ba7922ac2730be4d7f7500af984c447d4"}
Jul 07 18:23:47.260338 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:47.260063    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:47.284227 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:47.284181    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fbx64" podStartSLOduration=17.696779227 podStartE2EDuration="20.284167853s" totalImagesPullingTime="2.587388626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:47.284090405 +0000 UTC m=+54.859854300" watchObservedRunningTime="2026-07-07 18:23:47.284167853 +0000 UTC m=+54.859931746"
Jul 07 18:23:53.180915 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:53.180889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx7xc"
Jul 07 18:23:56.112189 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.112158    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d9f4cdd64-m72g2"]
Jul 07 18:23:56.295530 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.295503    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86f84896bf-jp45t"]
Jul 07 18:23:56.298515 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.298499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.368899 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.368838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dde10661-c01b-4718-86b6-18ee7cea2ca0-ca-trust-extracted\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.368899 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.368870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dde10661-c01b-4718-86b6-18ee7cea2ca0-registry-certificates\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.368899 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.368887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7vb\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-kube-api-access-cd7vb\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.369074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.368968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-bound-sa-token\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.369074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.368991    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dde10661-c01b-4718-86b6-18ee7cea2ca0-trusted-ca\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.369074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.369033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dde10661-c01b-4718-86b6-18ee7cea2ca0-installation-pull-secrets\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.369166 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.369075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-registry-tls\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.369166 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.369115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dde10661-c01b-4718-86b6-18ee7cea2ca0-image-registry-private-configuration\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.453415 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.453388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f84896bf-jp45t"]
Jul 07 18:23:56.469546 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dde10661-c01b-4718-86b6-18ee7cea2ca0-registry-certificates\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7vb\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-kube-api-access-cd7vb\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-bound-sa-token\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dde10661-c01b-4718-86b6-18ee7cea2ca0-trusted-ca\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dde10661-c01b-4718-86b6-18ee7cea2ca0-installation-pull-secrets\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469881 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-registry-tls\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469881 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dde10661-c01b-4718-86b6-18ee7cea2ca0-image-registry-private-configuration\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.469881 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.469792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dde10661-c01b-4718-86b6-18ee7cea2ca0-ca-trust-extracted\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.470161 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.470131    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dde10661-c01b-4718-86b6-18ee7cea2ca0-ca-trust-extracted\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.470414 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.470386    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dde10661-c01b-4718-86b6-18ee7cea2ca0-registry-certificates\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.470567 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.470547    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dde10661-c01b-4718-86b6-18ee7cea2ca0-trusted-ca\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.472010 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.471987    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dde10661-c01b-4718-86b6-18ee7cea2ca0-installation-pull-secrets\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.472181 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.472160    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dde10661-c01b-4718-86b6-18ee7cea2ca0-image-registry-private-configuration\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.472222 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.472168    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-registry-tls\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.496339 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.496311    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7vb\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-kube-api-access-cd7vb\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.496879 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.496859    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dde10661-c01b-4718-86b6-18ee7cea2ca0-bound-sa-token\") pod \"image-registry-86f84896bf-jp45t\" (UID: \"dde10661-c01b-4718-86b6-18ee7cea2ca0\") " pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.607127 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.607105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:56.825821 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:56.825792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f84896bf-jp45t"]
Jul 07 18:23:56.826949 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:56.826924    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde10661_c01b_4718_86b6_18ee7cea2ca0.slice/crio-e37af044a56393402ea5bbfb5d0eb524a6599b9d50bbd7565ac2c4a286a30524 WatchSource:0}: Error finding container e37af044a56393402ea5bbfb5d0eb524a6599b9d50bbd7565ac2c4a286a30524: Status 404 returned error can't find the container with id e37af044a56393402ea5bbfb5d0eb524a6599b9d50bbd7565ac2c4a286a30524
Jul 07 18:23:57.183849 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.183814    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"]
Jul 07 18:23:57.186809 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.186785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:23:57.189891 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.189856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 18:23:57.190195 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.190165    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7zfwp\""
Jul 07 18:23:57.203236 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.203213    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"]
Jul 07 18:23:57.265154 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.265128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fbx64"
Jul 07 18:23:57.276919 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.276898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e82a9d5-9214-46f9-a872-e62a8c1daf25-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-gh5v2\" (UID: \"7e82a9d5-9214-46f9-a872-e62a8c1daf25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:23:57.292387 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.292364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f84896bf-jp45t" event={"ID":"dde10661-c01b-4718-86b6-18ee7cea2ca0","Type":"ContainerStarted","Data":"09e1e5516a602fc888934ff151eb1d91c0397cce6821736fecfb462708aaf4cd"}
Jul 07 18:23:57.292522 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.292403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f84896bf-jp45t" event={"ID":"dde10661-c01b-4718-86b6-18ee7cea2ca0","Type":"ContainerStarted","Data":"e37af044a56393402ea5bbfb5d0eb524a6599b9d50bbd7565ac2c4a286a30524"}
Jul 07 18:23:57.292522 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.292480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:23:57.319765 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.319727    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86f84896bf-jp45t" podStartSLOduration=1.3197114079999999 podStartE2EDuration="1.319711408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:57.319542289 +0000 UTC m=+64.895306208" watchObservedRunningTime="2026-07-07 18:23:57.319711408 +0000 UTC m=+64.895475319"
Jul 07 18:23:57.378064 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.378036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e82a9d5-9214-46f9-a872-e62a8c1daf25-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-gh5v2\" (UID: \"7e82a9d5-9214-46f9-a872-e62a8c1daf25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:23:57.378550 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:57.378175    2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 18:23:57.378550 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:23:57.378240    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e82a9d5-9214-46f9-a872-e62a8c1daf25-tls-certificates podName:7e82a9d5-9214-46f9-a872-e62a8c1daf25 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:57.87822187 +0000 UTC m=+65.453985746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7e82a9d5-9214-46f9-a872-e62a8c1daf25-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-gh5v2" (UID: "7e82a9d5-9214-46f9-a872-e62a8c1daf25") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 18:23:57.881592 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.881559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e82a9d5-9214-46f9-a872-e62a8c1daf25-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-gh5v2\" (UID: \"7e82a9d5-9214-46f9-a872-e62a8c1daf25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:23:57.883903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:57.883884    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7e82a9d5-9214-46f9-a872-e62a8c1daf25-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-gh5v2\" (UID: \"7e82a9d5-9214-46f9-a872-e62a8c1daf25\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:23:58.095841 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.095813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:23:58.219537 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.219512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"]
Jul 07 18:23:58.222297 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:58.222272    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e82a9d5_9214_46f9_a872_e62a8c1daf25.slice/crio-04f74fd96d5a1d746ca6092f8174b0cdee24abc8df065fbec731188db96962b4 WatchSource:0}: Error finding container 04f74fd96d5a1d746ca6092f8174b0cdee24abc8df065fbec731188db96962b4: Status 404 returned error can't find the container with id 04f74fd96d5a1d746ca6092f8174b0cdee24abc8df065fbec731188db96962b4
Jul 07 18:23:58.296258 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.296233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2" event={"ID":"7e82a9d5-9214-46f9-a872-e62a8c1daf25","Type":"ContainerStarted","Data":"04f74fd96d5a1d746ca6092f8174b0cdee24abc8df065fbec731188db96962b4"}
Jul 07 18:23:58.586866 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.586840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:58.586986 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.586903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:58.590214 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.590196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 18:23:58.591175 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.591162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 18:23:58.600135 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.600107    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7e28baa-d67e-4cb0-b1f4-7f38600ea365-original-pull-secret\") pod \"global-pull-secret-syncer-tvpcl\" (UID: \"b7e28baa-d67e-4cb0-b1f4-7f38600ea365\") " pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:58.600604 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.600588    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c-metrics-certs\") pod \"network-metrics-daemon-6mmxv\" (UID: \"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c\") " pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:58.688002 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.687971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:58.690612 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.690595    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 18:23:58.700572 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.700552    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 18:23:58.710797 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.710768    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwxq\" (UniqueName: \"kubernetes.io/projected/7b130aef-ae7c-4338-bf24-234e14252958-kube-api-access-5gwxq\") pod \"network-check-target-mrffv\" (UID: \"7b130aef-ae7c-4338-bf24-234e14252958\") " pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:58.762373 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.762354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dxsjk\""
Jul 07 18:23:58.766509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.766497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvpcl"
Jul 07 18:23:58.769988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.769970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:23:58.775318 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.775301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wxf5l\""
Jul 07 18:23:58.783527 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.783506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mmxv"
Jul 07 18:23:58.908972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:58.908944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvpcl"]
Jul 07 18:23:58.912613 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:58.912586    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e28baa_d67e_4cb0_b1f4_7f38600ea365.slice/crio-e26e2d68cc16f9d758bc8ddaac7a98492e0ae260c41d39ad095c0faf21f24c15 WatchSource:0}: Error finding container e26e2d68cc16f9d758bc8ddaac7a98492e0ae260c41d39ad095c0faf21f24c15: Status 404 returned error can't find the container with id e26e2d68cc16f9d758bc8ddaac7a98492e0ae260c41d39ad095c0faf21f24c15
Jul 07 18:23:59.129602 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:59.129510    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6mmxv"]
Jul 07 18:23:59.132964 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:59.132935    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a2dfeb_a52d_4cfd_a47c_611ffca91a5c.slice/crio-a1fe5ee0316e7ccc0606c13d2fe3a4e2222ad7add3f6bd97bd3578c89e8ba8af WatchSource:0}: Error finding container a1fe5ee0316e7ccc0606c13d2fe3a4e2222ad7add3f6bd97bd3578c89e8ba8af: Status 404 returned error can't find the container with id a1fe5ee0316e7ccc0606c13d2fe3a4e2222ad7add3f6bd97bd3578c89e8ba8af
Jul 07 18:23:59.139992 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:59.139939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrffv"]
Jul 07 18:23:59.142408 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:23:59.142384    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b130aef_ae7c_4338_bf24_234e14252958.slice/crio-d443fa307c56b30ebcda735cf67348cf59cb71d07babd3299eccdf9a705ae538 WatchSource:0}: Error finding container d443fa307c56b30ebcda735cf67348cf59cb71d07babd3299eccdf9a705ae538: Status 404 returned error can't find the container with id d443fa307c56b30ebcda735cf67348cf59cb71d07babd3299eccdf9a705ae538
Jul 07 18:23:59.301510 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:59.301477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6mmxv" event={"ID":"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c","Type":"ContainerStarted","Data":"a1fe5ee0316e7ccc0606c13d2fe3a4e2222ad7add3f6bd97bd3578c89e8ba8af"}
Jul 07 18:23:59.302722 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:59.302700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrffv" event={"ID":"7b130aef-ae7c-4338-bf24-234e14252958","Type":"ContainerStarted","Data":"d443fa307c56b30ebcda735cf67348cf59cb71d07babd3299eccdf9a705ae538"}
Jul 07 18:23:59.303875 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:23:59.303853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvpcl" event={"ID":"b7e28baa-d67e-4cb0-b1f4-7f38600ea365","Type":"ContainerStarted","Data":"e26e2d68cc16f9d758bc8ddaac7a98492e0ae260c41d39ad095c0faf21f24c15"}
Jul 07 18:24:00.309277 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:00.309239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2" event={"ID":"7e82a9d5-9214-46f9-a872-e62a8c1daf25","Type":"ContainerStarted","Data":"54c1f108402017154be513c49af843fe65a53424fb9d8157cac89a33ed229b19"}
Jul 07 18:24:00.309752 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:00.309485    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:24:00.316080 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:00.316034    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2"
Jul 07 18:24:00.332631 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:00.332580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-gh5v2" podStartSLOduration=1.8846149890000001 podStartE2EDuration="3.33256269s" totalImagesPullingTime="1.447947701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:00.33145289 +0000 UTC m=+67.907216785" watchObservedRunningTime="2026-07-07 18:24:00.33256269 +0000 UTC m=+67.908326585"
Jul 07 18:24:01.276500 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.276469    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"]
Jul 07 18:24:01.279930 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.279903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.282356 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.282303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 18:24:01.283604 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.283583    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 18:24:01.283726 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.283632    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 18:24:01.283793 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.283775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k4jrf\""
Jul 07 18:24:01.283905 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.283889    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 18:24:01.283966 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.283921    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 18:24:01.294585 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.294561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"]
Jul 07 18:24:01.316529 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.316500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6mmxv" event={"ID":"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c","Type":"ContainerStarted","Data":"ff51955a45750ab6ac4373787fca0f096882803118f6877b07c89228857995f3"}
Jul 07 18:24:01.316882 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.316555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6mmxv" event={"ID":"b9a2dfeb-a52d-4cfd-a47c-611ffca91a5c","Type":"ContainerStarted","Data":"eb15a30979c221236125219600b65cf2c77a7632ef4594ae69b1228f075e1540"}
Jul 07 18:24:01.343001 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.342954    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6mmxv" podStartSLOduration=66.743406878 podStartE2EDuration="1m8.342936057s" totalImagesPullingTime="1.599529179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:01.34137252 +0000 UTC m=+68.917136406" watchObservedRunningTime="2026-07-07 18:24:01.342936057 +0000 UTC m=+68.918699955"
Jul 07 18:24:01.414040 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.414011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.414184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.414098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.414298 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.414279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.414359 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.414320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdx7\" (UniqueName: \"kubernetes.io/projected/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-kube-api-access-bkdx7\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.514709 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.514671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.514901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.514717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdx7\" (UniqueName: \"kubernetes.io/projected/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-kube-api-access-bkdx7\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.514901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.514748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.514901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.514798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.515480 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.515435    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.517289 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.517267    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.517374 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.517270    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.525293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.525271    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdx7\" (UniqueName: \"kubernetes.io/projected/a2fb5bcb-c48d-47cd-b063-88ea579aa5c6-kube-api-access-bkdx7\") pod \"prometheus-operator-567f7b8b7c-wzd9c\" (UID: \"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.593165 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.593134    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"
Jul 07 18:24:01.742930 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.742864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c"]
Jul 07 18:24:01.745918 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:01.745882    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fb5bcb_c48d_47cd_b063_88ea579aa5c6.slice/crio-72da4766f7d702e74b30198fc2ff0deea6b3dc96efc3b7028034db3373e502d4 WatchSource:0}: Error finding container 72da4766f7d702e74b30198fc2ff0deea6b3dc96efc3b7028034db3373e502d4: Status 404 returned error can't find the container with id 72da4766f7d702e74b30198fc2ff0deea6b3dc96efc3b7028034db3373e502d4
Jul 07 18:24:01.918405 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.918325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:24:01.920812 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:01.920789    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1faf637a-833d-4e33-8902-66cee03aa98a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b885r\" (UID: \"1faf637a-833d-4e33-8902-66cee03aa98a\") " pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:24:02.093347 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:02.093314    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-62q56\""
Jul 07 18:24:02.101511 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:02.101484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b885r"
Jul 07 18:24:02.321904 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:02.321856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c" event={"ID":"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6","Type":"ContainerStarted","Data":"72da4766f7d702e74b30198fc2ff0deea6b3dc96efc3b7028034db3373e502d4"}
Jul 07 18:24:02.939771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:02.939752    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b885r"]
Jul 07 18:24:02.942790 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:02.942761    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faf637a_833d_4e33_8902_66cee03aa98a.slice/crio-ed6982c80648bdbe2ad227f524619e496085fb81e21069646a69d232c540bc4d WatchSource:0}: Error finding container ed6982c80648bdbe2ad227f524619e496085fb81e21069646a69d232c540bc4d: Status 404 returned error can't find the container with id ed6982c80648bdbe2ad227f524619e496085fb81e21069646a69d232c540bc4d
Jul 07 18:24:03.327188 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:03.327136    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrffv" event={"ID":"7b130aef-ae7c-4338-bf24-234e14252958","Type":"ContainerStarted","Data":"8a435632a5254dee828b7a098795b163721ccf9cc69fb2e3a7beb97351a248af"}
Jul 07 18:24:03.327760 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:03.327241    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:24:03.328795 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:03.328768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b885r" event={"ID":"1faf637a-833d-4e33-8902-66cee03aa98a","Type":"ContainerStarted","Data":"ed6982c80648bdbe2ad227f524619e496085fb81e21069646a69d232c540bc4d"}
Jul 07 18:24:03.348156 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:03.347777    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mrffv" podStartSLOduration=66.625952365 podStartE2EDuration="1m10.347760072s" totalImagesPullingTime="3.721807707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:03.347430612 +0000 UTC m=+70.923194527" watchObservedRunningTime="2026-07-07 18:24:03.347760072 +0000 UTC m=+70.923523967"
Jul 07 18:24:06.119361 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:06.119329    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:24:06.341837 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:06.341797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvpcl" event={"ID":"b7e28baa-d67e-4cb0-b1f4-7f38600ea365","Type":"ContainerStarted","Data":"c470beb59b7c95d2b591c6d2e301d54bcc26559eb19790a93d4d0a88b20a8deb"}
Jul 07 18:24:06.343939 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:06.343913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c" event={"ID":"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6","Type":"ContainerStarted","Data":"1163bcba057e5e5bcb3feb3add0cb5a3dbeb963e5183536e1e7a4fac779d016c"}
Jul 07 18:24:06.344058 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:06.343960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c" event={"ID":"a2fb5bcb-c48d-47cd-b063-88ea579aa5c6","Type":"ContainerStarted","Data":"59420f97b8f01d1b83deedee1ca15e59d1b752bbf45da87106903abce7800609"}
Jul 07 18:24:06.376632 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:06.376563    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tvpcl" podStartSLOduration=66.930982712 podStartE2EDuration="1m13.376548393s" totalImagesPullingTime="6.445565681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:06.376251206 +0000 UTC m=+73.952015100" watchObservedRunningTime="2026-07-07 18:24:06.376548393 +0000 UTC m=+73.952312264"
Jul 07 18:24:08.732324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.732262    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wzd9c" podStartSLOduration=4.123870767 podStartE2EDuration="7.732240084s" totalImagesPullingTime="3.608369317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:06.401340535 +0000 UTC m=+73.977104430" watchObservedRunningTime="2026-07-07 18:24:08.732240084 +0000 UTC m=+76.308003978"
Jul 07 18:24:08.734486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.734429    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mgqxd"]
Jul 07 18:24:08.744201 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.744180    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"]
Jul 07 18:24:08.744339 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.744322    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.747002 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.746971    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 18:24:08.747002 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.746989    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 18:24:08.747308 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.747281    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j2kzd\""
Jul 07 18:24:08.747529 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.747322    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 18:24:08.747847 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.747825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.752918 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.752900    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hjt4m\""
Jul 07 18:24:08.753107 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.753078    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 18:24:08.753210 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.753192    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 18:24:08.753295 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.753197    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 18:24:08.754715 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.754656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"]
Jul 07 18:24:08.871504 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.871683 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.871683 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-root\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.871683 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-wtmp\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.871683 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.871683 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c69045-eaa2-447a-84c6-9ae74eff2017-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.871972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.871972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8wd6\" (UniqueName: \"kubernetes.io/projected/e766d398-d48a-4d6f-915e-e93087dfa0bd-kube-api-access-w8wd6\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.871972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-sys\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.871972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/14c69045-eaa2-447a-84c6-9ae74eff2017-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.871972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4g8\" (UniqueName: \"kubernetes.io/projected/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-api-access-lh4g8\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.871972 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-textfile\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.872257 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.871984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-tls\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.872257 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.872003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e766d398-d48a-4d6f-915e-e93087dfa0bd-metrics-client-ca\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.872257 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.872022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.973315 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.973506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8wd6\" (UniqueName: \"kubernetes.io/projected/e766d398-d48a-4d6f-915e-e93087dfa0bd-kube-api-access-w8wd6\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-sys\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/14c69045-eaa2-447a-84c6-9ae74eff2017-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.973506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4g8\" (UniqueName: \"kubernetes.io/projected/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-api-access-lh4g8\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.973506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-textfile\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-tls\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e766d398-d48a-4d6f-915e-e93087dfa0bd-metrics-client-ca\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-root\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-wtmp\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973728    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-sys\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.973832 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.973758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c69045-eaa2-447a-84c6-9ae74eff2017-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.974258 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974097    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/14c69045-eaa2-447a-84c6-9ae74eff2017-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.974258 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974176    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-root\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.974402 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974380    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-textfile\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.974493 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974473    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c69045-eaa2-447a-84c6-9ae74eff2017-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.974558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974506    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e766d398-d48a-4d6f-915e-e93087dfa0bd-metrics-client-ca\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.974656 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974624    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-wtmp\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.974774 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974720    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.974879 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.974830    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.976910 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.976865    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.977360 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.977336    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:08.977430 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.977398    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.977651 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.977632    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e766d398-d48a-4d6f-915e-e93087dfa0bd-node-exporter-tls\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.983336 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.983281    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8wd6\" (UniqueName: \"kubernetes.io/projected/e766d398-d48a-4d6f-915e-e93087dfa0bd-kube-api-access-w8wd6\") pod \"node-exporter-mgqxd\" (UID: \"e766d398-d48a-4d6f-915e-e93087dfa0bd\") " pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:08.984121 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:08.984098    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4g8\" (UniqueName: \"kubernetes.io/projected/14c69045-eaa2-447a-84c6-9ae74eff2017-kube-api-access-lh4g8\") pod \"kube-state-metrics-7c9dbb8b7d-plhh8\" (UID: \"14c69045-eaa2-447a-84c6-9ae74eff2017\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:09.057194 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:09.057161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgqxd"
Jul 07 18:24:09.065036 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:09.065015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"
Jul 07 18:24:09.123636 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:09.123605    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode766d398_d48a_4d6f_915e_e93087dfa0bd.slice/crio-b0aceaa86d656b735a0db181367640f9cfdc5c6817b80dc545082f67d057facf WatchSource:0}: Error finding container b0aceaa86d656b735a0db181367640f9cfdc5c6817b80dc545082f67d057facf: Status 404 returned error can't find the container with id b0aceaa86d656b735a0db181367640f9cfdc5c6817b80dc545082f67d057facf
Jul 07 18:24:09.245167 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:09.245107    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8"]
Jul 07 18:24:09.247787 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:09.247760    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c69045_eaa2_447a_84c6_9ae74eff2017.slice/crio-171619e4b78cd2b10ca657ead38caa29b58dfa8e4f8ba88408ddbefbc0c8a4bc WatchSource:0}: Error finding container 171619e4b78cd2b10ca657ead38caa29b58dfa8e4f8ba88408ddbefbc0c8a4bc: Status 404 returned error can't find the container with id 171619e4b78cd2b10ca657ead38caa29b58dfa8e4f8ba88408ddbefbc0c8a4bc
Jul 07 18:24:09.355711 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:09.355676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b885r" event={"ID":"1faf637a-833d-4e33-8902-66cee03aa98a","Type":"ContainerStarted","Data":"ec0931c3d884ed0fe152bac2da0796d04570dbc7d29312194cc5419766fb7b4d"}
Jul 07 18:24:09.356799 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:09.356773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8" event={"ID":"14c69045-eaa2-447a-84c6-9ae74eff2017","Type":"ContainerStarted","Data":"171619e4b78cd2b10ca657ead38caa29b58dfa8e4f8ba88408ddbefbc0c8a4bc"}
Jul 07 18:24:09.357859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:09.357837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgqxd" event={"ID":"e766d398-d48a-4d6f-915e-e93087dfa0bd","Type":"ContainerStarted","Data":"b0aceaa86d656b735a0db181367640f9cfdc5c6817b80dc545082f67d057facf"}
Jul 07 18:24:10.365145 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:10.365088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b885r" event={"ID":"1faf637a-833d-4e33-8902-66cee03aa98a","Type":"ContainerStarted","Data":"8ed264442530558d505d536095216671c2eacd87145d604bdeca82b6ca1ed5b0"}
Jul 07 18:24:10.367201 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:10.367169    2565 generic.go:356] "Generic (PLEG): container finished" podID="e766d398-d48a-4d6f-915e-e93087dfa0bd" containerID="434f252e569838ecf11c821dfcb4105f859a0e195cbe1c3066b57f9afbb72f1c" exitCode=0
Jul 07 18:24:10.367327 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:10.367208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgqxd" event={"ID":"e766d398-d48a-4d6f-915e-e93087dfa0bd","Type":"ContainerDied","Data":"434f252e569838ecf11c821dfcb4105f859a0e195cbe1c3066b57f9afbb72f1c"}
Jul 07 18:24:11.374907 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:11.374865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgqxd" event={"ID":"e766d398-d48a-4d6f-915e-e93087dfa0bd","Type":"ContainerStarted","Data":"273e506ec84a2a3c8281b730eb7f728a8a8ab14f34353d583c906bf4edac1abb"}
Jul 07 18:24:11.375353 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:11.374927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgqxd" event={"ID":"e766d398-d48a-4d6f-915e-e93087dfa0bd","Type":"ContainerStarted","Data":"915a201bb610c2138bee7ce93f99f81cde89c696921fc700f2e5f540422bc8a5"}
Jul 07 18:24:11.377421 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:11.377381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8" event={"ID":"14c69045-eaa2-447a-84c6-9ae74eff2017","Type":"ContainerStarted","Data":"291b230042832fbf657adb78943d8ed375d3502dbc17db93e172d81f92fb33e2"}
Jul 07 18:24:11.377577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:11.377449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8" event={"ID":"14c69045-eaa2-447a-84c6-9ae74eff2017","Type":"ContainerStarted","Data":"f758fae8c83587145a0396994bdc399d4b97c852aff3062c08f32cc8ad74a2a2"}
Jul 07 18:24:11.398319 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:11.398271    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mgqxd" podStartSLOduration=2.272438343 podStartE2EDuration="3.398256563s" totalImagesPullingTime="1.054034566s" totalInitContainerRuntime="71.783654ms" isStatefulPod=true podCreationTimestamp="2026-07-07 18:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:11.396745849 +0000 UTC m=+78.972509801" watchObservedRunningTime="2026-07-07 18:24:11.398256563 +0000 UTC m=+78.974020484"
Jul 07 18:24:13.386806 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.386764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b885r" event={"ID":"1faf637a-833d-4e33-8902-66cee03aa98a","Type":"ContainerStarted","Data":"f797b84dcd0c1dc1acc918be802e84aa6d5cb78370eaac0b07042b67bbd58b71"}
Jul 07 18:24:13.388872 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.388847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8" event={"ID":"14c69045-eaa2-447a-84c6-9ae74eff2017","Type":"ContainerStarted","Data":"ddf026d9232db9cf2692ccf8705fcef2e723a4ecc63747e9d30e87b7f792d018"}
Jul 07 18:24:13.405101 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.405076    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"]
Jul 07 18:24:13.407144 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.407127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.409776 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.409755    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 18:24:13.409898 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.409880    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r69sz\""
Jul 07 18:24:13.415580 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.415540    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b885r" podStartSLOduration=35.040976702 podStartE2EDuration="44.415522944s" totalImagesPullingTime="9.374546242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:23:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:13.414681793 +0000 UTC m=+80.990445700" watchObservedRunningTime="2026-07-07 18:24:13.415522944 +0000 UTC m=+80.991286839"
Jul 07 18:24:13.419035 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.419013    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"]
Jul 07 18:24:13.443643 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.443595    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-plhh8" podStartSLOduration=3.515752877 podStartE2EDuration="5.443580574s" totalImagesPullingTime="1.927827697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:13.442259703 +0000 UTC m=+81.018023596" watchObservedRunningTime="2026-07-07 18:24:13.443580574 +0000 UTC m=+81.019344468"
Jul 07 18:24:13.510302 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.510274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54accd2e-5b7f-46a7-b477-9250a42fa77b-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f9hcc\" (UID: \"54accd2e-5b7f-46a7-b477-9250a42fa77b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.510449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.510392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnqq\" (UniqueName: \"kubernetes.io/projected/54accd2e-5b7f-46a7-b477-9250a42fa77b-kube-api-access-zdnqq\") pod \"monitoring-plugin-d5dcff7cf-f9hcc\" (UID: \"54accd2e-5b7f-46a7-b477-9250a42fa77b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.611089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.611068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54accd2e-5b7f-46a7-b477-9250a42fa77b-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f9hcc\" (UID: \"54accd2e-5b7f-46a7-b477-9250a42fa77b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.611190 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.611102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnqq\" (UniqueName: \"kubernetes.io/projected/54accd2e-5b7f-46a7-b477-9250a42fa77b-kube-api-access-zdnqq\") pod \"monitoring-plugin-d5dcff7cf-f9hcc\" (UID: \"54accd2e-5b7f-46a7-b477-9250a42fa77b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.613484 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.613436    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54accd2e-5b7f-46a7-b477-9250a42fa77b-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f9hcc\" (UID: \"54accd2e-5b7f-46a7-b477-9250a42fa77b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.620051 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.620030    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnqq\" (UniqueName: \"kubernetes.io/projected/54accd2e-5b7f-46a7-b477-9250a42fa77b-kube-api-access-zdnqq\") pod \"monitoring-plugin-d5dcff7cf-f9hcc\" (UID: \"54accd2e-5b7f-46a7-b477-9250a42fa77b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.716480 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.716386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:13.843893 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:13.843822    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"]
Jul 07 18:24:13.846338 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:13.846306    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54accd2e_5b7f_46a7_b477_9250a42fa77b.slice/crio-4089ba8192634d29014a05a66e4af6eedf2eb00e0671de374a347ff301392911 WatchSource:0}: Error finding container 4089ba8192634d29014a05a66e4af6eedf2eb00e0671de374a347ff301392911: Status 404 returned error can't find the container with id 4089ba8192634d29014a05a66e4af6eedf2eb00e0671de374a347ff301392911
Jul 07 18:24:14.393318 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.393277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc" event={"ID":"54accd2e-5b7f-46a7-b477-9250a42fa77b","Type":"ContainerStarted","Data":"4089ba8192634d29014a05a66e4af6eedf2eb00e0671de374a347ff301392911"}
Jul 07 18:24:14.774186 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.774054    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-dl8gt"]
Jul 07 18:24:14.777055 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.777031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:14.779899 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.779874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 18:24:14.780048 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.779965    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 18:24:14.780473 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.780432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-sfljp\""
Jul 07 18:24:14.809623 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.809597    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-dl8gt"]
Jul 07 18:24:14.923548 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:14.923512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwl8s\" (UniqueName: \"kubernetes.io/projected/f1ece3eb-0fdc-447a-b0db-be45265df4a3-kube-api-access-xwl8s\") pod \"downloads-685bfb5f64-dl8gt\" (UID: \"f1ece3eb-0fdc-447a-b0db-be45265df4a3\") " pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:15.024597 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.024519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwl8s\" (UniqueName: \"kubernetes.io/projected/f1ece3eb-0fdc-447a-b0db-be45265df4a3-kube-api-access-xwl8s\") pod \"downloads-685bfb5f64-dl8gt\" (UID: \"f1ece3eb-0fdc-447a-b0db-be45265df4a3\") " pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:15.082927 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.082904    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwl8s\" (UniqueName: \"kubernetes.io/projected/f1ece3eb-0fdc-447a-b0db-be45265df4a3-kube-api-access-xwl8s\") pod \"downloads-685bfb5f64-dl8gt\" (UID: \"f1ece3eb-0fdc-447a-b0db-be45265df4a3\") " pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:15.090133 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.090108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:15.202249 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.202150    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:24:15.207037 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.207005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.213635 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.213395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 18:24:15.213635 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.213395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dpcz8\""
Jul 07 18:24:15.213635 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.213395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 18:24:15.213635 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.213400    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 18:24:15.214729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.214638    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 18:24:15.214729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.214637    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-72mkg4o0vpbg2\""
Jul 07 18:24:15.214729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.214664    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 18:24:15.214729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.214702    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 18:24:15.215803 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.215768    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 18:24:15.216489 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.216279    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 18:24:15.218854 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.216884    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 18:24:15.220971 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.220940    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 18:24:15.221746 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.221727    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 18:24:15.222156 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.222139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 18:24:15.222282 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.222219    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 18:24:15.225107 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.225065    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 18:24:15.227214 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.227196    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 18:24:15.245562 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.245528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:24:15.327399 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327581 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327581 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-config\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327581 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327581 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327805 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327805 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327805 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327805 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327805 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.327805 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-config-out\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-web-config\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.327970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.328000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328405 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.328160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328405 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.328215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmrv\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-kube-api-access-tcmrv\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.328405 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.328242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.429693 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.429693 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-config-out\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-web-config\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmrv\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-kube-api-access-tcmrv\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.429978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-config\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.430212 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.431049 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.431049 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.430925    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.431400    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.431571    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.431970    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.432654    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.434114    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.434378    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.434659    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.434778    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.435272    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.435553    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-config\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436621 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.436274    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.436675 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.436642    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.437692 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.437647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-config-out\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.438095 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.437941    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.438631 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.438588    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.439009 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.438949    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.439873 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.439849    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-web-config\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.441068 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.441025    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmrv\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-kube-api-access-tcmrv\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.441258 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.441214    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.524270 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.524235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:15.649193 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.649147    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-dl8gt"]
Jul 07 18:24:15.653530 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:15.653450    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ece3eb_0fdc_447a_b0db_be45265df4a3.slice/crio-795fcafb43261da06195cf160edb0a27d98add88bb9169eac6648c588cc886cd WatchSource:0}: Error finding container 795fcafb43261da06195cf160edb0a27d98add88bb9169eac6648c588cc886cd: Status 404 returned error can't find the container with id 795fcafb43261da06195cf160edb0a27d98add88bb9169eac6648c588cc886cd
Jul 07 18:24:15.726329 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:15.726299    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:24:15.729262 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:24:15.729236    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f2d595_505f_41bd_9988_728e2160ff93.slice/crio-e0cf09400eebc5286d11cd4452e273933e67b773c9d135bfea288fc0816e7923 WatchSource:0}: Error finding container e0cf09400eebc5286d11cd4452e273933e67b773c9d135bfea288fc0816e7923: Status 404 returned error can't find the container with id e0cf09400eebc5286d11cd4452e273933e67b773c9d135bfea288fc0816e7923
Jul 07 18:24:16.403492 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:16.403441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc" event={"ID":"54accd2e-5b7f-46a7-b477-9250a42fa77b","Type":"ContainerStarted","Data":"c0e531bfa641160b26de583447a3004a0639092edc8b9a39cbfec5608900c8f8"}
Jul 07 18:24:16.403720 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:16.403520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:16.405369 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:16.405335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"e0cf09400eebc5286d11cd4452e273933e67b773c9d135bfea288fc0816e7923"}
Jul 07 18:24:16.407048 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:16.407010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-dl8gt" event={"ID":"f1ece3eb-0fdc-447a-b0db-be45265df4a3","Type":"ContainerStarted","Data":"795fcafb43261da06195cf160edb0a27d98add88bb9169eac6648c588cc886cd"}
Jul 07 18:24:16.409981 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:16.409956    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc"
Jul 07 18:24:16.439084 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:16.439037    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f9hcc" podStartSLOduration=1.7616293029999999 podStartE2EDuration="3.439024282s" totalImagesPullingTime="1.677394979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:16.437532168 +0000 UTC m=+84.013296074" watchObservedRunningTime="2026-07-07 18:24:16.439024282 +0000 UTC m=+84.014788176"
Jul 07 18:24:17.413920 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:17.413882    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61" exitCode=0
Jul 07 18:24:17.414098 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:17.413976    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"}
Jul 07 18:24:18.303255 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:18.303224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86f84896bf-jp45t"
Jul 07 18:24:21.138114 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.138048    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2" podUID="b93285c2-1bc0-4c94-8490-207d1ed3072a" containerName="registry" containerID="cri-o://038246bd2ea90ca7154253be0181c3b7633fdcd44f121e430b074b8d212b5b1f" gracePeriod=30
Jul 07 18:24:21.433088 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.433053    2565 generic.go:356] "Generic (PLEG): container finished" podID="b93285c2-1bc0-4c94-8490-207d1ed3072a" containerID="038246bd2ea90ca7154253be0181c3b7633fdcd44f121e430b074b8d212b5b1f" exitCode=0
Jul 07 18:24:21.433200 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.433107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2" event={"ID":"b93285c2-1bc0-4c94-8490-207d1ed3072a","Type":"ContainerDied","Data":"038246bd2ea90ca7154253be0181c3b7633fdcd44f121e430b074b8d212b5b1f"}
Jul 07 18:24:21.465352 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.465087    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:24:21.587558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-certificates\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.587779 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587587    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-image-registry-private-configuration\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.587779 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.587779 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587652    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8ls8\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-kube-api-access-s8ls8\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.587779 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-bound-sa-token\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.587779 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-trusted-ca\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.588090 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93285c2-1bc0-4c94-8490-207d1ed3072a-ca-trust-extracted\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.588090 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.587878    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-installation-pull-secrets\") pod \"b93285c2-1bc0-4c94-8490-207d1ed3072a\" (UID: \"b93285c2-1bc0-4c94-8490-207d1ed3072a\") "
Jul 07 18:24:21.588771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.588587    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:24:21.588771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.588614    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:24:21.591024 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.590980    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:24:21.591024 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.590990    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:24:21.591024 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.591020    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:24:21.591159 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.591136    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-kube-api-access-s8ls8" (OuterVolumeSpecName: "kube-api-access-s8ls8") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "kube-api-access-s8ls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:24:21.591204 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.591182    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:24:21.597911 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.597888    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93285c2-1bc0-4c94-8490-207d1ed3072a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b93285c2-1bc0-4c94-8490-207d1ed3072a" (UID: "b93285c2-1bc0-4c94-8490-207d1ed3072a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689494    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-installation-pull-secrets\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689524    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-certificates\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689534    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93285c2-1bc0-4c94-8490-207d1ed3072a-image-registry-private-configuration\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689543    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-registry-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689551    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8ls8\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-kube-api-access-s8ls8\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689560    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93285c2-1bc0-4c94-8490-207d1ed3072a-bound-sa-token\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689568    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93285c2-1bc0-4c94-8490-207d1ed3072a-trusted-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:21.689577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:21.689576    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93285c2-1bc0-4c94-8490-207d1ed3072a-ca-trust-extracted\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:24:22.439496 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.439451    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2"
Jul 07 18:24:22.440058 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.439496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9f4cdd64-m72g2" event={"ID":"b93285c2-1bc0-4c94-8490-207d1ed3072a","Type":"ContainerDied","Data":"80b069410d65ad8233a9b6faccd7e3b525ea82e16731ca7d099c470898fe1be7"}
Jul 07 18:24:22.440058 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.439586    2565 scope.go:117] "RemoveContainer" containerID="038246bd2ea90ca7154253be0181c3b7633fdcd44f121e430b074b8d212b5b1f"
Jul 07 18:24:22.446086 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.446057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"}
Jul 07 18:24:22.446215 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.446179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"}
Jul 07 18:24:22.469928 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.469900    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d9f4cdd64-m72g2"]
Jul 07 18:24:22.472864 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:22.472843    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d9f4cdd64-m72g2"]
Jul 07 18:24:23.048108 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:23.048072    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93285c2-1bc0-4c94-8490-207d1ed3072a" path="/var/lib/kubelet/pods/b93285c2-1bc0-4c94-8490-207d1ed3072a/volumes"
Jul 07 18:24:24.465546 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:24.465498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"}
Jul 07 18:24:24.466011 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:24.465622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"}
Jul 07 18:24:24.466011 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:24.465666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"}
Jul 07 18:24:24.466011 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:24.465708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerStarted","Data":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"}
Jul 07 18:24:24.507919 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:24.507864    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.2593476089999998 podStartE2EDuration="9.507848478s" totalImagesPullingTime="7.222685047s" totalInitContainerRuntime="25.815822ms" isStatefulPod=false podCreationTimestamp="2026-07-07 18:24:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:24.505265176 +0000 UTC m=+92.081029152" watchObservedRunningTime="2026-07-07 18:24:24.507848478 +0000 UTC m=+92.083612349"
Jul 07 18:24:25.524544 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:25.524514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:24:34.335469 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:34.335430    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mrffv"
Jul 07 18:24:38.525477 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:38.525427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-dl8gt" event={"ID":"f1ece3eb-0fdc-447a-b0db-be45265df4a3","Type":"ContainerStarted","Data":"b35b04b79e7e4064879886266e1401b59e36dcb3067e3ba603424f039a08ada5"}
Jul 07 18:24:38.525996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:38.525515    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:38.539993 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:38.539968    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-dl8gt"
Jul 07 18:24:38.560340 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:24:38.560292    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-dl8gt" podStartSLOduration=2.753484696 podStartE2EDuration="24.560276905s" totalImagesPullingTime="21.806792209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:38.558678543 +0000 UTC m=+106.134442436" watchObservedRunningTime="2026-07-07 18:24:38.560276905 +0000 UTC m=+106.136040832"
Jul 07 18:25:15.525192 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:15.525157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:15.599094 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:15.599069    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:15.684762 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:15.684736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:33.942771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.942736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:25:33.943253 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.943167    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="prometheus" containerID="cri-o://c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288" gracePeriod=600
Jul 07 18:25:33.943253 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.943182    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy" containerID="cri-o://c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c" gracePeriod=600
Jul 07 18:25:33.943253 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.943205    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-web" containerID="cri-o://d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855" gracePeriod=600
Jul 07 18:25:33.943415 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.943205    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="thanos-sidecar" containerID="cri-o://cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302" gracePeriod=600
Jul 07 18:25:33.943415 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.943251    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-thanos" containerID="cri-o://654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56" gracePeriod=600
Jul 07 18:25:33.943415 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:33.943272    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="config-reloader" containerID="cri-o://68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a" gracePeriod=600
Jul 07 18:25:34.202561 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.202519    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:34.295355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295573 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-1\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295653 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-config-out\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295709 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-trusted-ca-bundle\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295757 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295720    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-thanos-prometheus-http-client-file\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295806 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295762    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcmrv\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-kube-api-access-tcmrv\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295806 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295797    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-metrics-client-certs\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-web-config\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295852    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-tls\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-db\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.295903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295882    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.296089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-config\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295958    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-metrics-client-ca\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.295986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-serving-certs-ca-bundle\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296076    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-tls-assets\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-kubelet-serving-ca-bundle\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296142    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-kube-rbac-proxy\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-grpc-tls\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296183    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.296343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-0\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296343 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-2\") pod \"62f2d595-505f-41bd-9988-728e2160ff93\" (UID: \"62f2d595-505f-41bd-9988-728e2160ff93\") "
Jul 07 18:25:34.296670 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296572    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.296670 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296592    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.297906 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.296870    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.297906 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.297169    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.297906 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.297300    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.297906 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.297589    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.298324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.298286    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:25:34.299573 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.299530    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.300721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.300686    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.300913 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.300886    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.300913 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.300902    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:25:34.301376 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.301345    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:25:34.301871 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.301846    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-kube-api-access-tcmrv" (OuterVolumeSpecName: "kube-api-access-tcmrv") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "kube-api-access-tcmrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:25:34.302254 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.302218    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.302373 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.302282    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.302373 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.302341    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.303096 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.303065    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.303192 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.303122    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-config-out" (OuterVolumeSpecName: "config-out") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:25:34.303611 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.303583    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-config" (OuterVolumeSpecName: "config") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.312396 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.312368    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-web-config" (OuterVolumeSpecName: "web-config") pod "62f2d595-505f-41bd-9988-728e2160ff93" (UID: "62f2d595-505f-41bd-9988-728e2160ff93"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:25:34.397541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397506    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-metrics-client-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397535    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397559    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397571    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-tls-assets\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397580    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397589    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-kube-rbac-proxy\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397598    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-grpc-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397606    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397614    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397623    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397631    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-config-out\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397639    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397647    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmrv\" (UniqueName: \"kubernetes.io/projected/62f2d595-505f-41bd-9988-728e2160ff93-kube-api-access-tcmrv\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397655    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-metrics-client-certs\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397664    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-web-config\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.397671 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397672    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.398106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397680    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62f2d595-505f-41bd-9988-728e2160ff93-prometheus-k8s-db\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.398106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.397689    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/62f2d595-505f-41bd-9988-728e2160ff93-config\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:25:34.751102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751067    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56" exitCode=0
Jul 07 18:25:34.751102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751091    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c" exitCode=0
Jul 07 18:25:34.751102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751101    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855" exitCode=0
Jul 07 18:25:34.751102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751109    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302" exitCode=0
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751116    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a" exitCode=0
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751123    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f2d595-505f-41bd-9988-728e2160ff93" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288" exitCode=0
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751146    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"}
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"}
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751346    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"}
Jul 07 18:25:34.751382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751363    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.751721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"}
Jul 07 18:25:34.751721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"}
Jul 07 18:25:34.751721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"}
Jul 07 18:25:34.751721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.751527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62f2d595-505f-41bd-9988-728e2160ff93","Type":"ContainerDied","Data":"e0cf09400eebc5286d11cd4452e273933e67b773c9d135bfea288fc0816e7923"}
Jul 07 18:25:34.758987 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.758969    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.765517 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.765499    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.771780 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.771763    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.777770 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.777755    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.783901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.783884    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.790824 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.790808    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.796996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.796979    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.797254 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.797234    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.797300 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797267    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"} err="failed to get container status \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist"
Jul 07 18:25:34.797300 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797298    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.797550 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.797531    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.797609 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797554    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"} err="failed to get container status \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist"
Jul 07 18:25:34.797609 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797569    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.797767 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.797752    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.797801 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797771    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"} err="failed to get container status \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist"
Jul 07 18:25:34.797801 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797785    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.797996 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.797981    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.798044 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.797999    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"} err="failed to get container status \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist"
Jul 07 18:25:34.798044 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798011    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.798199 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.798184    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.798238 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798201    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"} err="failed to get container status \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist"
Jul 07 18:25:34.798238 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798213    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.798418 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.798402    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.798483 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798420    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"} err="failed to get container status \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist"
Jul 07 18:25:34.798483 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798431    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.798671 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:25:34.798651    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.798724 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798676    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"} err="failed to get container status \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist"
Jul 07 18:25:34.798724 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798704    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.798947 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798930    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"} err="failed to get container status \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist"
Jul 07 18:25:34.798993 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.798947    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.799158 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799137    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"} err="failed to get container status \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist"
Jul 07 18:25:34.799202 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799160    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.799346 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799331    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"} err="failed to get container status \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist"
Jul 07 18:25:34.799387 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799346    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.799586 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799568    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"} err="failed to get container status \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist"
Jul 07 18:25:34.799623 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799587    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.799833 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799815    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"} err="failed to get container status \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist"
Jul 07 18:25:34.799881 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.799834    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.800031 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800010    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"} err="failed to get container status \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist"
Jul 07 18:25:34.800072 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800032    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.800208 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800193    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"} err="failed to get container status \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist"
Jul 07 18:25:34.800248 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800207    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.800379 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800364    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"} err="failed to get container status \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist"
Jul 07 18:25:34.800425 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800380    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.800604 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800584    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"} err="failed to get container status \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist"
Jul 07 18:25:34.800651 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800606    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.800833 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800814    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"} err="failed to get container status \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist"
Jul 07 18:25:34.800880 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.800833    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.801043 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801027    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"} err="failed to get container status \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist"
Jul 07 18:25:34.801085 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801043    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.801245 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801227    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"} err="failed to get container status \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist"
Jul 07 18:25:34.801295 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801245    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.801483 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801449    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"} err="failed to get container status \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist"
Jul 07 18:25:34.801536 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801483    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.801730 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801711    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"} err="failed to get container status \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist"
Jul 07 18:25:34.801798 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801731    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.801953 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801930    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"} err="failed to get container status \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist"
Jul 07 18:25:34.802001 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.801955    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.802163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802145    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"} err="failed to get container status \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist"
Jul 07 18:25:34.802207 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802164    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.802370 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802350    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"} err="failed to get container status \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist"
Jul 07 18:25:34.802440 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802372    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.802614 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802598    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"} err="failed to get container status \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist"
Jul 07 18:25:34.802674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802614    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.802841 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802823    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"} err="failed to get container status \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist"
Jul 07 18:25:34.802890 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.802842    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.803027 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803013    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"} err="failed to get container status \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist"
Jul 07 18:25:34.803027 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803025    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.803222 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803208    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"} err="failed to get container status \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist"
Jul 07 18:25:34.803263 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803222    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.803412 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803394    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"} err="failed to get container status \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist"
Jul 07 18:25:34.803480 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803413    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.803753 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803733    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"} err="failed to get container status \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist"
Jul 07 18:25:34.803798 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803754    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.803943 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803924    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"} err="failed to get container status \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist"
Jul 07 18:25:34.803983 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.803944    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.804119 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804100    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"} err="failed to get container status \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist"
Jul 07 18:25:34.804184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804120    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.804333 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804315    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"} err="failed to get container status \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist"
Jul 07 18:25:34.804382 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804334    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.804525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804506    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"} err="failed to get container status \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist"
Jul 07 18:25:34.804596 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804527    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.804709 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804695    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"} err="failed to get container status \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist"
Jul 07 18:25:34.804754 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804711    2565 scope.go:117] "RemoveContainer" containerID="654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"
Jul 07 18:25:34.804863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804845    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56"} err="failed to get container status \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": rpc error: code = NotFound desc = could not find container \"654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56\": container with ID starting with 654b06d2dffa038cff57c7905818b5697a0e9b488eb6dd57ebb207733f0daa56 not found: ID does not exist"
Jul 07 18:25:34.804863 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.804862    2565 scope.go:117] "RemoveContainer" containerID="c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"
Jul 07 18:25:34.805086 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805067    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c"} err="failed to get container status \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": rpc error: code = NotFound desc = could not find container \"c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c\": container with ID starting with c25082ac9d5f2bec7389ebba790a9c773af5e3fb8841ddac9e1f3f170615885c not found: ID does not exist"
Jul 07 18:25:34.805135 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805087    2565 scope.go:117] "RemoveContainer" containerID="d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"
Jul 07 18:25:34.805308 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805292    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855"} err="failed to get container status \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": rpc error: code = NotFound desc = could not find container \"d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855\": container with ID starting with d1e162bd12c279c0ed3d97e4725a0948c6a0b638afe8e7490e871e3633a9d855 not found: ID does not exist"
Jul 07 18:25:34.805363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805307    2565 scope.go:117] "RemoveContainer" containerID="cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"
Jul 07 18:25:34.805524 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805509    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302"} err="failed to get container status \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": rpc error: code = NotFound desc = could not find container \"cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302\": container with ID starting with cfea56ad1022ada14b18cd11799965cd8d771a6b5b799c8bcb5ea100537eb302 not found: ID does not exist"
Jul 07 18:25:34.805583 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805523    2565 scope.go:117] "RemoveContainer" containerID="68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"
Jul 07 18:25:34.805687 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805674    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a"} err="failed to get container status \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": rpc error: code = NotFound desc = could not find container \"68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a\": container with ID starting with 68e76d8a9f544f9340b2254ca71b6df1a282e90d04af724a64e0559e36187c1a not found: ID does not exist"
Jul 07 18:25:34.805733 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805687    2565 scope.go:117] "RemoveContainer" containerID="c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"
Jul 07 18:25:34.805848 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805834    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288"} err="failed to get container status \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": rpc error: code = NotFound desc = could not find container \"c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288\": container with ID starting with c8b331fc7bf7036256329277c3c7a99efa33d9511d646215d0214e4bf89f6288 not found: ID does not exist"
Jul 07 18:25:34.805898 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.805847    2565 scope.go:117] "RemoveContainer" containerID="c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"
Jul 07 18:25:34.806019 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.806006    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61"} err="failed to get container status \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": rpc error: code = NotFound desc = could not find container \"c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61\": container with ID starting with c22a9d598f477364f7daa12d3cb5378179e387d85046ef06972239d90b1bdb61 not found: ID does not exist"
Jul 07 18:25:34.809668 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.809650    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:25:34.817058 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.817037    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:25:34.903492 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:25:34.903788 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903772    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="init-config-reloader"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903791    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="init-config-reloader"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903806    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-thanos"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903814    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-thanos"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903836    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903850    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="config-reloader"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903859    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="config-reloader"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903873    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-web"
Jul 07 18:25:34.903877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903881    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-web"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903896    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93285c2-1bc0-4c94-8490-207d1ed3072a" containerName="registry"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903904    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93285c2-1bc0-4c94-8490-207d1ed3072a" containerName="registry"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903916    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="thanos-sidecar"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903925    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="thanos-sidecar"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="prometheus"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.903946    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="prometheus"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904011    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-thanos"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904024    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="prometheus"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904040    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="thanos-sidecar"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904049    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="config-reloader"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904057    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93285c2-1bc0-4c94-8490-207d1ed3072a" containerName="registry"
Jul 07 18:25:34.904294 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.904066    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f2d595-505f-41bd-9988-728e2160ff93" containerName="kube-rbac-proxy-web"
Jul 07 18:25:34.908107 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.908087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:34.912402 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.912379    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 18:25:34.912549 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.912402    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 18:25:34.913592 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.913576    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 18:25:34.913850 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.913836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-72mkg4o0vpbg2\""
Jul 07 18:25:34.914356 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.914339    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 18:25:34.914440 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.914339    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 18:25:34.914440 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.914339    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 18:25:34.914708 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.914685    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dpcz8\""
Jul 07 18:25:34.914814 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.914686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 18:25:34.915330 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.915313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 18:25:34.915927 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.915717    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 18:25:34.915927 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.915815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 18:25:34.915927 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.915823    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 18:25:34.918507 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.918487    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 18:25:34.919062 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.918958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 18:25:34.923428 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.923407    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 18:25:34.924657 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.924642    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 18:25:34.963083 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:34.963054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-config\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8s7\" (UniqueName: \"kubernetes.io/projected/b94bee8c-19cc-4163-badc-981755cbe606-kube-api-access-gb8s7\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-web-config\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b94bee8c-19cc-4163-badc-981755cbe606-config-out\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b94bee8c-19cc-4163-badc-981755cbe606-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.001949 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.001751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.046308 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.046276    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f2d595-505f-41bd-9988-728e2160ff93" path="/var/lib/kubelet/pods/62f2d595-505f-41bd-9988-728e2160ff93/volumes"
Jul 07 18:25:35.103050 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103194 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b94bee8c-19cc-4163-badc-981755cbe606-config-out\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103256 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b94bee8c-19cc-4163-badc-981755cbe606-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103256 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103808 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103821    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.103901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.103993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-config\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8s7\" (UniqueName: \"kubernetes.io/projected/b94bee8c-19cc-4163-badc-981755cbe606-kube-api-access-gb8s7\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-web-config\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104490 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104490 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104635 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104611    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.104999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.104971    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.106645 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.105518    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.106645 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.106286    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.106645 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.106330    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.106870 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.106795    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b94bee8c-19cc-4163-badc-981755cbe606-config-out\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.107676 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.107008    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.107676 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.107059    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.107676 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.107187    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b94bee8c-19cc-4163-badc-981755cbe606-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.107676 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.107426    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.107676 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.107623    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.107988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.107821    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b94bee8c-19cc-4163-badc-981755cbe606-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.108495 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.108425    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.108701 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.108683    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.108958 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.108932    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-config\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.109037 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.108982    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.109104 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.109085    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.110324 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.110307    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b94bee8c-19cc-4163-badc-981755cbe606-web-config\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.120222 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.120201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8s7\" (UniqueName: \"kubernetes.io/projected/b94bee8c-19cc-4163-badc-981755cbe606-kube-api-access-gb8s7\") pod \"prometheus-k8s-0\" (UID: \"b94bee8c-19cc-4163-badc-981755cbe606\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.219615 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.219577    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:25:35.361851 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:25:35.361821    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94bee8c_19cc_4163_badc_981755cbe606.slice/crio-49567797316f2843e0f40320436edad41ae15d9ef479687d3b0f64d427cb8605 WatchSource:0}: Error finding container 49567797316f2843e0f40320436edad41ae15d9ef479687d3b0f64d427cb8605: Status 404 returned error can't find the container with id 49567797316f2843e0f40320436edad41ae15d9ef479687d3b0f64d427cb8605
Jul 07 18:25:35.363107 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.363083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 18:25:35.757278 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.757246    2565 generic.go:356] "Generic (PLEG): container finished" podID="b94bee8c-19cc-4163-badc-981755cbe606" containerID="f6171d753d24c806f8b32f9d598e6f57138e97b85f8d0e1234a6abf38ddc4f48" exitCode=0
Jul 07 18:25:35.757470 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.757298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerDied","Data":"f6171d753d24c806f8b32f9d598e6f57138e97b85f8d0e1234a6abf38ddc4f48"}
Jul 07 18:25:35.757470 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:35.757350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"49567797316f2843e0f40320436edad41ae15d9ef479687d3b0f64d427cb8605"}
Jul 07 18:25:36.768424 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.768393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"1de54f170c50f9b306ed319e19bf463973a0b438246840f292d671f10348afec"}
Jul 07 18:25:36.768825 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.768499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"a3bf3e8c540be90a9ec764e05b1cbb297c58d3f5e007c20013500b7874eb2e02"}
Jul 07 18:25:36.768825 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.768529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"7cab6a04519d2e902416fdc0f8b11c5ba4ba4fe13e23e973e6ab19222ba1e2d7"}
Jul 07 18:25:36.768825 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.768552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"ee076e160e87e0f60f76871c7f49994faa2deb7e5ee4c528d1fbc2c1340da640"}
Jul 07 18:25:36.768825 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.768576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"cad735d5156328d28c74c0f67341a7fbaf185d5c86a9adf85e9e8b8604522136"}
Jul 07 18:25:36.768825 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.768600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94bee8c-19cc-4163-badc-981755cbe606","Type":"ContainerStarted","Data":"b6e9ddebb76d9d205b40453ff95bbb0c2a799cfe5e6a3c0f6db2f63c7168e8f8"}
Jul 07 18:25:36.806260 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:36.806121    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.786699242 podStartE2EDuration="2.806099176s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.399934ms" isStatefulPod=false podCreationTimestamp="2026-07-07 18:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:25:36.803743369 +0000 UTC m=+164.379507307" watchObservedRunningTime="2026-07-07 18:25:36.806099176 +0000 UTC m=+164.381863071"
Jul 07 18:25:40.220599 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:25:40.220564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:26:35.219954 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:26:35.219910    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:26:35.235304 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:26:35.235278    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:26:36.000366 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:26:36.000336    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 18:27:52.787155 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:27:52.787128    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 18:28:41.074164 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.074133    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mdt5m"]
Jul 07 18:28:41.076586 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.076569    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.080956 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.080930    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 18:28:41.081173 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.081157    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 18:28:41.081303 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.081286    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-plv9s\""
Jul 07 18:28:41.089251 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.089231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mdt5m"]
Jul 07 18:28:41.196955 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.196929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/274fdfff-e6ef-4169-b189-a1959cd57853-bound-sa-token\") pod \"cert-manager-55b4d954d5-mdt5m\" (UID: \"274fdfff-e6ef-4169-b189-a1959cd57853\") " pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.196955 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.196957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszkr\" (UniqueName: \"kubernetes.io/projected/274fdfff-e6ef-4169-b189-a1959cd57853-kube-api-access-rszkr\") pod \"cert-manager-55b4d954d5-mdt5m\" (UID: \"274fdfff-e6ef-4169-b189-a1959cd57853\") " pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.297397 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.297360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/274fdfff-e6ef-4169-b189-a1959cd57853-bound-sa-token\") pod \"cert-manager-55b4d954d5-mdt5m\" (UID: \"274fdfff-e6ef-4169-b189-a1959cd57853\") " pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.297397 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.297397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszkr\" (UniqueName: \"kubernetes.io/projected/274fdfff-e6ef-4169-b189-a1959cd57853-kube-api-access-rszkr\") pod \"cert-manager-55b4d954d5-mdt5m\" (UID: \"274fdfff-e6ef-4169-b189-a1959cd57853\") " pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.306936 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.306887    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/274fdfff-e6ef-4169-b189-a1959cd57853-bound-sa-token\") pod \"cert-manager-55b4d954d5-mdt5m\" (UID: \"274fdfff-e6ef-4169-b189-a1959cd57853\") " pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.307503 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.307481    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszkr\" (UniqueName: \"kubernetes.io/projected/274fdfff-e6ef-4169-b189-a1959cd57853-kube-api-access-rszkr\") pod \"cert-manager-55b4d954d5-mdt5m\" (UID: \"274fdfff-e6ef-4169-b189-a1959cd57853\") " pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.396991 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.396892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mdt5m"
Jul 07 18:28:41.527379 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.527348    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mdt5m"]
Jul 07 18:28:41.530811 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:28:41.530785    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274fdfff_e6ef_4169_b189_a1959cd57853.slice/crio-cacad2160be1ab1d92fe5d42fcd171ebe09dab4fcf2fdba1d35256ce9011409e WatchSource:0}: Error finding container cacad2160be1ab1d92fe5d42fcd171ebe09dab4fcf2fdba1d35256ce9011409e: Status 404 returned error can't find the container with id cacad2160be1ab1d92fe5d42fcd171ebe09dab4fcf2fdba1d35256ce9011409e
Jul 07 18:28:41.532683 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:41.532663    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:28:42.435010 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:42.434967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mdt5m" event={"ID":"274fdfff-e6ef-4169-b189-a1959cd57853","Type":"ContainerStarted","Data":"cacad2160be1ab1d92fe5d42fcd171ebe09dab4fcf2fdba1d35256ce9011409e"}
Jul 07 18:28:44.445742 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:44.445705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mdt5m" event={"ID":"274fdfff-e6ef-4169-b189-a1959cd57853","Type":"ContainerStarted","Data":"2f75586ce156551c2ba7fa40cb9ddce48926d7547d89fa12423634b093b67d81"}
Jul 07 18:28:44.463543 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:28:44.463497    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mdt5m" podStartSLOduration=0.637451734 podStartE2EDuration="3.463481181s" totalImagesPullingTime="2.826029447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:28:44.461840051 +0000 UTC m=+352.037603948" watchObservedRunningTime="2026-07-07 18:28:44.463481181 +0000 UTC m=+352.039245068"
Jul 07 18:29:02.770579 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.770550    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"]
Jul 07 18:29:02.773093 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.773074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.776978 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.776931    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 18:29:02.776978 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.776951    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 18:29:02.777192 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.777064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-llbm9\""
Jul 07 18:29:02.777768 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.777754    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 18:29:02.778116 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.778099    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 18:29:02.803585 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.803563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"]
Jul 07 18:29:02.841617 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.841593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-tmp\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.841716 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.841622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-manifests\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.841716 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.841648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-webhook-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.841785 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.841734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.841785 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.841761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptpr\" (UniqueName: \"kubernetes.io/projected/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-kube-api-access-tptpr\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.942837 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.942812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.942939 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.942839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptpr\" (UniqueName: \"kubernetes.io/projected/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-kube-api-access-tptpr\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.942939 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.942893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-tmp\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.943016 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.942988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-manifests\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.943066 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.943024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-webhook-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.943328 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.943310    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-manifests\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.945189 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.945128    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-tmp\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.945289 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.945214    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-webhook-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.945289 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.945259    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:02.951200 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:02.951180    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptpr\" (UniqueName: \"kubernetes.io/projected/09aefc4d-ab66-40a0-b4f4-41c5ec59e286-kube-api-access-tptpr\") pod \"opendatahub-operator-controller-manager-7595c859c8-w6rmf\" (UID: \"09aefc4d-ab66-40a0-b4f4-41c5ec59e286\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:03.083771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:03.083732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:03.215660 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:03.215637    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"]
Jul 07 18:29:03.218209 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:29:03.218177    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09aefc4d_ab66_40a0_b4f4_41c5ec59e286.slice/crio-8e30b3218b136e68da797ef75b1a46dee6fce54ddd19861c167c170aa98034fd WatchSource:0}: Error finding container 8e30b3218b136e68da797ef75b1a46dee6fce54ddd19861c167c170aa98034fd: Status 404 returned error can't find the container with id 8e30b3218b136e68da797ef75b1a46dee6fce54ddd19861c167c170aa98034fd
Jul 07 18:29:03.519044 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:03.518952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf" event={"ID":"09aefc4d-ab66-40a0-b4f4-41c5ec59e286","Type":"ContainerStarted","Data":"8e30b3218b136e68da797ef75b1a46dee6fce54ddd19861c167c170aa98034fd"}
Jul 07 18:29:06.534674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:06.534601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf" event={"ID":"09aefc4d-ab66-40a0-b4f4-41c5ec59e286","Type":"ContainerStarted","Data":"2bcc7f09dba856d01922514d1474417f36787b589090f7570b0a71ab4e9f1ec7"}
Jul 07 18:29:07.539397 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:07.539325    2565 generic.go:356] "Generic (PLEG): container finished" podID="09aefc4d-ab66-40a0-b4f4-41c5ec59e286" containerID="2bcc7f09dba856d01922514d1474417f36787b589090f7570b0a71ab4e9f1ec7" exitCode=0
Jul 07 18:29:07.539397 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:07.539376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf" event={"ID":"09aefc4d-ab66-40a0-b4f4-41c5ec59e286","Type":"ContainerDied","Data":"2bcc7f09dba856d01922514d1474417f36787b589090f7570b0a71ab4e9f1ec7"}
Jul 07 18:29:08.545494 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:08.545436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf" event={"ID":"09aefc4d-ab66-40a0-b4f4-41c5ec59e286","Type":"ContainerStarted","Data":"0bb676e9c866b003f5d9a63d6b31ed2495f9049a7133ac76c4db9332f5a752af"}
Jul 07 18:29:08.545860 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:08.545544    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:08.622631 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:08.622580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf" podStartSLOduration=2.313255963 podStartE2EDuration="6.622563005s" totalImagesPullingTime="3.255019479s" totalInitContainerRuntime="1.054287563s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:08.621757813 +0000 UTC m=+376.197521726" watchObservedRunningTime="2026-07-07 18:29:08.622563005 +0000 UTC m=+376.198326899"
Jul 07 18:29:19.551028 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:19.550997    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-w6rmf"
Jul 07 18:29:22.682767 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.682732    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"]
Jul 07 18:29:22.690166 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.690145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.694001 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.693977    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 18:29:22.694139 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.694012    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 18:29:22.694139 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.694061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-msk2x\""
Jul 07 18:29:22.694139 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.694015    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 18:29:22.694139 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.694018    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 18:29:22.694313 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.694204    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 18:29:22.698898 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.698871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"]
Jul 07 18:29:22.788738 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.788711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd3114e-9650-4f88-a055-18fcdff1e2e4-cert\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.788738 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.788740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9bd3114e-9650-4f88-a055-18fcdff1e2e4-manager-config\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.788883 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.788772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjwr\" (UniqueName: \"kubernetes.io/projected/9bd3114e-9650-4f88-a055-18fcdff1e2e4-kube-api-access-sxjwr\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.788883 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.788863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bd3114e-9650-4f88-a055-18fcdff1e2e4-metrics-cert\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.889338 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.889317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bd3114e-9650-4f88-a055-18fcdff1e2e4-metrics-cert\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.889449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.889359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd3114e-9650-4f88-a055-18fcdff1e2e4-cert\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.889449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.889377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9bd3114e-9650-4f88-a055-18fcdff1e2e4-manager-config\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.889449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.889412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjwr\" (UniqueName: \"kubernetes.io/projected/9bd3114e-9650-4f88-a055-18fcdff1e2e4-kube-api-access-sxjwr\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.889997 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.889975    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9bd3114e-9650-4f88-a055-18fcdff1e2e4-manager-config\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.891646 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.891629    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bd3114e-9650-4f88-a055-18fcdff1e2e4-metrics-cert\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.891773 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.891700    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd3114e-9650-4f88-a055-18fcdff1e2e4-cert\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.917323 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.917296    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjwr\" (UniqueName: \"kubernetes.io/projected/9bd3114e-9650-4f88-a055-18fcdff1e2e4-kube-api-access-sxjwr\") pod \"lws-controller-manager-749c476b6d-bfq9t\" (UID: \"9bd3114e-9650-4f88-a055-18fcdff1e2e4\") " pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:22.999981 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:22.999923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:23.149478 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:23.149422    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"]
Jul 07 18:29:23.152444 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:29:23.152416    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd3114e_9650_4f88_a055_18fcdff1e2e4.slice/crio-c05e3d672693dd1210e6628bc43d96b5a67493f3916feac6f6fb8671505d7e72 WatchSource:0}: Error finding container c05e3d672693dd1210e6628bc43d96b5a67493f3916feac6f6fb8671505d7e72: Status 404 returned error can't find the container with id c05e3d672693dd1210e6628bc43d96b5a67493f3916feac6f6fb8671505d7e72
Jul 07 18:29:23.605903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:23.605868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t" event={"ID":"9bd3114e-9650-4f88-a055-18fcdff1e2e4","Type":"ContainerStarted","Data":"c05e3d672693dd1210e6628bc43d96b5a67493f3916feac6f6fb8671505d7e72"}
Jul 07 18:29:25.549656 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.549611    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"]
Jul 07 18:29:25.552202 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.552179    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.554687 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.554664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 18:29:25.555139 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.555121    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7srkw\""
Jul 07 18:29:25.555797 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.555772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 18:29:25.556041 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.556023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 18:29:25.556120 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.556072    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 18:29:25.567784 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.567756    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"]
Jul 07 18:29:25.614093 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.614060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-tls-certs\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.614257 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.614107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrjl\" (UniqueName: \"kubernetes.io/projected/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-kube-api-access-frrjl\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.614257 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.614244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-tmp\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.715670 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.715628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-tmp\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.715846 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.715684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-tls-certs\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.715846 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.715826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrjl\" (UniqueName: \"kubernetes.io/projected/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-kube-api-access-frrjl\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.717987 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.717965    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-tmp\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.718088 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.718003    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-tls-certs\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.725259 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.725230    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrjl\" (UniqueName: \"kubernetes.io/projected/6c84fa22-38ff-4d6e-95f5-8b7a4fec781f-kube-api-access-frrjl\") pod \"kube-auth-proxy-f5865f688-mkbdd\" (UID: \"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.864798 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.864771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"
Jul 07 18:29:25.993113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:25.993078    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f5865f688-mkbdd"]
Jul 07 18:29:25.996891 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:29:25.996866    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c84fa22_38ff_4d6e_95f5_8b7a4fec781f.slice/crio-b2ec505f1047800b02c7aa3116cec869833232ccaa0d3520a5c4cddbefb3388a WatchSource:0}: Error finding container b2ec505f1047800b02c7aa3116cec869833232ccaa0d3520a5c4cddbefb3388a: Status 404 returned error can't find the container with id b2ec505f1047800b02c7aa3116cec869833232ccaa0d3520a5c4cddbefb3388a
Jul 07 18:29:26.621674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:26.621632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd" event={"ID":"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f","Type":"ContainerStarted","Data":"b2ec505f1047800b02c7aa3116cec869833232ccaa0d3520a5c4cddbefb3388a"}
Jul 07 18:29:26.623511 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:26.623478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t" event={"ID":"9bd3114e-9650-4f88-a055-18fcdff1e2e4","Type":"ContainerStarted","Data":"2dbe1a8fe966d821e85afa4dca7067f1ab99ced114eaf544c4755fba42f975ba"}
Jul 07 18:29:26.623997 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:26.623975    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:29:26.662663 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:26.662608    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t" podStartSLOduration=2.089121382 podStartE2EDuration="4.66259308s" totalImagesPullingTime="2.573471698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:26.662423004 +0000 UTC m=+394.238186916" watchObservedRunningTime="2026-07-07 18:29:26.66259308 +0000 UTC m=+394.238356972"
Jul 07 18:29:29.638178 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:29.638142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd" event={"ID":"6c84fa22-38ff-4d6e-95f5-8b7a4fec781f","Type":"ContainerStarted","Data":"a04d6fe97b49d8c13e11d9d2c0a4a39858836f066f26f4ef9329e0f1174e61dc"}
Jul 07 18:29:29.655494 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:29.655427    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f5865f688-mkbdd" podStartSLOduration=1.42947837 podStartE2EDuration="4.655410381s" totalImagesPullingTime="3.225932011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:29.654011844 +0000 UTC m=+397.229775739" watchObservedRunningTime="2026-07-07 18:29:29.655410381 +0000 UTC m=+397.231174274"
Jul 07 18:29:38.635369 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:29:38.635340    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-749c476b6d-bfq9t"
Jul 07 18:31:12.999119 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:12.999086    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"]
Jul 07 18:31:13.002569 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.002547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.005264 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.005237    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 07 18:31:13.005264 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.005261    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 07 18:31:13.006030 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.006014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 18:31:13.006085 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.006021    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 18:31:13.006085 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.006071    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z7bnh\""
Jul 07 18:31:13.013353 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.013331    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"]
Jul 07 18:31:13.131512 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.131485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3595e30a-3255-460a-8e9c-f1b4c14445f8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.131673 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.131535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95xs\" (UniqueName: \"kubernetes.io/projected/3595e30a-3255-460a-8e9c-f1b4c14445f8-kube-api-access-l95xs\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.131673 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.131612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3595e30a-3255-460a-8e9c-f1b4c14445f8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.232660 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.232624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3595e30a-3255-460a-8e9c-f1b4c14445f8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.232858 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.232685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95xs\" (UniqueName: \"kubernetes.io/projected/3595e30a-3255-460a-8e9c-f1b4c14445f8-kube-api-access-l95xs\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.232858 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.232729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3595e30a-3255-460a-8e9c-f1b4c14445f8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.232858 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:13.232840    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 07 18:31:13.233031 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:13.232928    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3595e30a-3255-460a-8e9c-f1b4c14445f8-plugin-serving-cert podName:3595e30a-3255-460a-8e9c-f1b4c14445f8 nodeName:}" failed. No retries permitted until 2026-07-07 18:31:13.732905654 +0000 UTC m=+501.308669529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3595e30a-3255-460a-8e9c-f1b4c14445f8-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-pwbwp" (UID: "3595e30a-3255-460a-8e9c-f1b4c14445f8") : secret "plugin-serving-cert" not found
Jul 07 18:31:13.233389 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.233367    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3595e30a-3255-460a-8e9c-f1b4c14445f8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.244177 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.244145    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95xs\" (UniqueName: \"kubernetes.io/projected/3595e30a-3255-460a-8e9c-f1b4c14445f8-kube-api-access-l95xs\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.737059 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.737023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3595e30a-3255-460a-8e9c-f1b4c14445f8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.739273 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.739251    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3595e30a-3255-460a-8e9c-f1b4c14445f8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pwbwp\" (UID: \"3595e30a-3255-460a-8e9c-f1b4c14445f8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:13.910960 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:13.910931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"
Jul 07 18:31:14.093814 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:14.093784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp"]
Jul 07 18:31:14.098307 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:31:14.098263    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3595e30a_3255_460a_8e9c_f1b4c14445f8.slice/crio-1068b68da3636d7e4a22807f0280e8bd9f2f3c83d875ba1f8867c6852f7bc1c2 WatchSource:0}: Error finding container 1068b68da3636d7e4a22807f0280e8bd9f2f3c83d875ba1f8867c6852f7bc1c2: Status 404 returned error can't find the container with id 1068b68da3636d7e4a22807f0280e8bd9f2f3c83d875ba1f8867c6852f7bc1c2
Jul 07 18:31:15.068695 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:15.068656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp" event={"ID":"3595e30a-3255-460a-8e9c-f1b4c14445f8","Type":"ContainerStarted","Data":"1068b68da3636d7e4a22807f0280e8bd9f2f3c83d875ba1f8867c6852f7bc1c2"}
Jul 07 18:31:23.302704 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.302671    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"]
Jul 07 18:31:23.306032 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.306011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.309357 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.309330    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2v2g\""
Jul 07 18:31:23.326984 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.326954    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"]
Jul 07 18:31:23.402730 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.402705    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"]
Jul 07 18:31:23.402992 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:23.402971    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-cwqzh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf" podUID="d790414c-b177-4de7-89a7-58b955353342"
Jul 07 18:31:23.422144 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.422118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/d790414c-b177-4de7-89a7-58b955353342-kube-api-access-cwqzh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.422250 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.422156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d790414c-b177-4de7-89a7-58b955353342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.523184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.523152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/d790414c-b177-4de7-89a7-58b955353342-kube-api-access-cwqzh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.523323 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.523197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d790414c-b177-4de7-89a7-58b955353342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.523580 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.523560    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d790414c-b177-4de7-89a7-58b955353342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.538012 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.537987    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/d790414c-b177-4de7-89a7-58b955353342-kube-api-access-cwqzh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:23.871895 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.871853    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"]
Jul 07 18:31:23.884007 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.883701    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"]
Jul 07 18:31:23.940232 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.940143    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"]
Jul 07 18:31:23.947550 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.947525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:23.959559 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:23.959517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"]
Jul 07 18:31:24.108866 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.108834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:24.111024 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:24.110984    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" is forbidden: User \"system:node:ip-10-0-130-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-31.ec2.internal' and this object" podUID="d790414c-b177-4de7-89a7-58b955353342" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:24.117273 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.117249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:24.119580 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:24.119549    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" is forbidden: User \"system:node:ip-10-0-130-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-31.ec2.internal' and this object" podUID="d790414c-b177-4de7-89a7-58b955353342" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:24.127995 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.127937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-74cz5\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.128102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.128021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l6rv\" (UniqueName: \"kubernetes.io/projected/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-kube-api-access-4l6rv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-74cz5\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.228890 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.228858    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/d790414c-b177-4de7-89a7-58b955353342-kube-api-access-cwqzh\") pod \"d790414c-b177-4de7-89a7-58b955353342\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") "
Jul 07 18:31:24.229066 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.228985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d790414c-b177-4de7-89a7-58b955353342-extensions-socket-volume\") pod \"d790414c-b177-4de7-89a7-58b955353342\" (UID: \"d790414c-b177-4de7-89a7-58b955353342\") "
Jul 07 18:31:24.229148 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.229123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l6rv\" (UniqueName: \"kubernetes.io/projected/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-kube-api-access-4l6rv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-74cz5\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.229261 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.229204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-74cz5\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.229316 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.229271    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d790414c-b177-4de7-89a7-58b955353342-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d790414c-b177-4de7-89a7-58b955353342" (UID: "d790414c-b177-4de7-89a7-58b955353342"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:31:24.229639 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.229614    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-74cz5\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.231224 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.231199    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d790414c-b177-4de7-89a7-58b955353342-kube-api-access-cwqzh" (OuterVolumeSpecName: "kube-api-access-cwqzh") pod "d790414c-b177-4de7-89a7-58b955353342" (UID: "d790414c-b177-4de7-89a7-58b955353342"). InnerVolumeSpecName "kube-api-access-cwqzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:31:24.246528 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.246500    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l6rv\" (UniqueName: \"kubernetes.io/projected/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-kube-api-access-4l6rv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-74cz5\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.263250 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.263226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:24.330602 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.330569    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d790414c-b177-4de7-89a7-58b955353342-extensions-socket-volume\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:31:24.330602 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:24.330603    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/d790414c-b177-4de7-89a7-58b955353342-kube-api-access-cwqzh\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:31:25.048049 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:25.048016    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d790414c-b177-4de7-89a7-58b955353342" path="/var/lib/kubelet/pods/d790414c-b177-4de7-89a7-58b955353342/volumes"
Jul 07 18:31:25.113421 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:25.113390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:25.117731 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:25.117699    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tz6pf\" is forbidden: User \"system:node:ip-10-0-130-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-31.ec2.internal' and this object" podUID="d790414c-b177-4de7-89a7-58b955353342" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tz6pf"
Jul 07 18:31:36.209309 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:36.209267    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"]
Jul 07 18:31:38.163836 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:38.163814    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"]
Jul 07 18:31:38.165536 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:31:38.165508    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a340a5_0896_497f_aa8e_9aa7c8c90a1f.slice/crio-eddbb83a46c4c9e20731eb8cd01583c81a11d9df1694b171cf76e6c597e6e0c2 WatchSource:0}: Error finding container eddbb83a46c4c9e20731eb8cd01583c81a11d9df1694b171cf76e6c597e6e0c2: Status 404 returned error can't find the container with id eddbb83a46c4c9e20731eb8cd01583c81a11d9df1694b171cf76e6c597e6e0c2
Jul 07 18:31:38.178968 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:38.178943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5" event={"ID":"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f","Type":"ContainerStarted","Data":"eddbb83a46c4c9e20731eb8cd01583c81a11d9df1694b171cf76e6c597e6e0c2"}
Jul 07 18:31:39.185006 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:39.184973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp" event={"ID":"3595e30a-3255-460a-8e9c-f1b4c14445f8","Type":"ContainerStarted","Data":"1c89189aaff5f325f6e9a00699e26648977a213db12cbed3129e18b2958d7cb8"}
Jul 07 18:31:39.204938 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:39.204887    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pwbwp" podStartSLOduration=2.869089286 podStartE2EDuration="27.204871086s" totalImagesPullingTime="24.3357818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:39.204071875 +0000 UTC m=+526.779835773" watchObservedRunningTime="2026-07-07 18:31:39.204871086 +0000 UTC m=+526.780634979"
Jul 07 18:31:46.215541 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.215506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5" event={"ID":"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f","Type":"ContainerStarted","Data":"5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610"}
Jul 07 18:31:46.216017 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.215560    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5" podUID="f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" containerName="manager" containerID="cri-o://5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610" gracePeriod=10
Jul 07 18:31:46.216017 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.215654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:46.236363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.236314    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5" podStartSLOduration=15.698048255 podStartE2EDuration="23.236297858s" totalImagesPullingTime="7.538249603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:46.233972096 +0000 UTC m=+533.809735987" watchObservedRunningTime="2026-07-07 18:31:46.236297858 +0000 UTC m=+533.812061752"
Jul 07 18:31:46.449178 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.449152    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:46.518136 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.518079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l6rv\" (UniqueName: \"kubernetes.io/projected/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-kube-api-access-4l6rv\") pod \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") "
Jul 07 18:31:46.518264 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.518139    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-extensions-socket-volume\") pod \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\" (UID: \"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f\") "
Jul 07 18:31:46.518614 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.518590    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" (UID: "f3a340a5-0896-497f-aa8e-9aa7c8c90a1f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:31:46.520180 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.520159    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-kube-api-access-4l6rv" (OuterVolumeSpecName: "kube-api-access-4l6rv") pod "f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" (UID: "f3a340a5-0896-497f-aa8e-9aa7c8c90a1f"). InnerVolumeSpecName "kube-api-access-4l6rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:31:46.619106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.619080    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-extensions-socket-volume\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:31:46.619106 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:46.619104    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l6rv\" (UniqueName: \"kubernetes.io/projected/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f-kube-api-access-4l6rv\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:31:47.221590 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.221552    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" containerID="5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610" exitCode=0
Jul 07 18:31:47.221981 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.221601    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"
Jul 07 18:31:47.221981 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.221625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5" event={"ID":"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f","Type":"ContainerDied","Data":"5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610"}
Jul 07 18:31:47.221981 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.221670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5" event={"ID":"f3a340a5-0896-497f-aa8e-9aa7c8c90a1f","Type":"ContainerDied","Data":"eddbb83a46c4c9e20731eb8cd01583c81a11d9df1694b171cf76e6c597e6e0c2"}
Jul 07 18:31:47.221981 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.221699    2565 scope.go:117] "RemoveContainer" containerID="5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610"
Jul 07 18:31:47.230598 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.230580    2565 scope.go:117] "RemoveContainer" containerID="5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610"
Jul 07 18:31:47.230883 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:31:47.230862    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610\": container with ID starting with 5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610 not found: ID does not exist" containerID="5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610"
Jul 07 18:31:47.230948 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.230891    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610"} err="failed to get container status \"5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610\": rpc error: code = NotFound desc = could not find container \"5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610\": container with ID starting with 5cfbb56c6cfc46412ed3485a705f798154a24174adf6513a40403ad1df683610 not found: ID does not exist"
Jul 07 18:31:47.242033 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.242013    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"]
Jul 07 18:31:47.245549 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:47.245528    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-74cz5"]
Jul 07 18:31:49.047087 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:31:49.047052    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" path="/var/lib/kubelet/pods/f3a340a5-0896-497f-aa8e-9aa7c8c90a1f/volumes"
Jul 07 18:32:07.050059 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.050026    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:07.050518 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.050328    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" containerName="manager"
Jul 07 18:32:07.050518 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.050339    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" containerName="manager"
Jul 07 18:32:07.050518 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.050418    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a340a5-0896-497f-aa8e-9aa7c8c90a1f" containerName="manager"
Jul 07 18:32:07.104626 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.104590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:07.104883 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.104713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.107758 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.107737    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 18:32:07.145051 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.145019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:07.182125 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.182087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/265f1c21-d4dc-4944-892b-8a683485db7e-config-file\") pod \"limitador-limitador-64c94fd64f-rckdl\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.182256 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.182162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjrm\" (UniqueName: \"kubernetes.io/projected/265f1c21-d4dc-4944-892b-8a683485db7e-kube-api-access-fqjrm\") pod \"limitador-limitador-64c94fd64f-rckdl\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.282804 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.282775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/265f1c21-d4dc-4944-892b-8a683485db7e-config-file\") pod \"limitador-limitador-64c94fd64f-rckdl\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.282927 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.282811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjrm\" (UniqueName: \"kubernetes.io/projected/265f1c21-d4dc-4944-892b-8a683485db7e-kube-api-access-fqjrm\") pod \"limitador-limitador-64c94fd64f-rckdl\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.283347 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.283327    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/265f1c21-d4dc-4944-892b-8a683485db7e-config-file\") pod \"limitador-limitador-64c94fd64f-rckdl\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.291927 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.291899    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjrm\" (UniqueName: \"kubernetes.io/projected/265f1c21-d4dc-4944-892b-8a683485db7e-kube-api-access-fqjrm\") pod \"limitador-limitador-64c94fd64f-rckdl\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.417358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.417329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:07.566318 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:07.566295    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:07.568740 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:32:07.568701    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265f1c21_d4dc_4944_892b_8a683485db7e.slice/crio-b789139fe5f1ce996f63b6a700d604c270d82b897b414e7360500b0f28994ffd WatchSource:0}: Error finding container b789139fe5f1ce996f63b6a700d604c270d82b897b414e7360500b0f28994ffd: Status 404 returned error can't find the container with id b789139fe5f1ce996f63b6a700d604c270d82b897b414e7360500b0f28994ffd
Jul 07 18:32:08.194756 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.194722    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gtbnq"]
Jul 07 18:32:08.202195 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.202166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:08.205389 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.205362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p868l\""
Jul 07 18:32:08.206412 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.206385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gtbnq"]
Jul 07 18:32:08.292530 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.292492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhjc\" (UniqueName: \"kubernetes.io/projected/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7-kube-api-access-wxhjc\") pod \"authorino-75d556bd77-gtbnq\" (UID: \"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7\") " pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:08.311758 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.311725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl" event={"ID":"265f1c21-d4dc-4944-892b-8a683485db7e","Type":"ContainerStarted","Data":"b789139fe5f1ce996f63b6a700d604c270d82b897b414e7360500b0f28994ffd"}
Jul 07 18:32:08.393891 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.393860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhjc\" (UniqueName: \"kubernetes.io/projected/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7-kube-api-access-wxhjc\") pod \"authorino-75d556bd77-gtbnq\" (UID: \"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7\") " pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:08.404185 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.404152    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhjc\" (UniqueName: \"kubernetes.io/projected/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7-kube-api-access-wxhjc\") pod \"authorino-75d556bd77-gtbnq\" (UID: \"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7\") " pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:08.511990 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.511660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:08.685774 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:08.685706    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gtbnq"]
Jul 07 18:32:08.690988 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:32:08.690956    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eaeaa5e_1feb_4851_aba9_7691ed7e64e7.slice/crio-adc68df8b52674d2af96e98ff023b79411ca209cc642f9ea16017c1ec3b9fec2 WatchSource:0}: Error finding container adc68df8b52674d2af96e98ff023b79411ca209cc642f9ea16017c1ec3b9fec2: Status 404 returned error can't find the container with id adc68df8b52674d2af96e98ff023b79411ca209cc642f9ea16017c1ec3b9fec2
Jul 07 18:32:09.321856 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:09.321816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gtbnq" event={"ID":"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7","Type":"ContainerStarted","Data":"adc68df8b52674d2af96e98ff023b79411ca209cc642f9ea16017c1ec3b9fec2"}
Jul 07 18:32:12.338262 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:12.338235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl" event={"ID":"265f1c21-d4dc-4944-892b-8a683485db7e","Type":"ContainerStarted","Data":"7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534"}
Jul 07 18:32:13.343536 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:13.343494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gtbnq" event={"ID":"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7","Type":"ContainerStarted","Data":"3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a"}
Jul 07 18:32:13.344003 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:13.343631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:13.361571 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:13.361521    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl" podStartSLOduration=1.696811955 podStartE2EDuration="6.361506572s" totalImagesPullingTime="4.664694617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:13.360341967 +0000 UTC m=+560.936105857" watchObservedRunningTime="2026-07-07 18:32:13.361506572 +0000 UTC m=+560.937270466"
Jul 07 18:32:13.377885 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:13.377829    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gtbnq" podStartSLOduration=1.795902345 podStartE2EDuration="5.377813704s" totalImagesPullingTime="3.581911359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:13.376876519 +0000 UTC m=+560.952640414" watchObservedRunningTime="2026-07-07 18:32:13.377813704 +0000 UTC m=+560.953577599"
Jul 07 18:32:24.348777 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:24.348744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:24.407979 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:24.407942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:24.408169 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:24.408122    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl" podUID="265f1c21-d4dc-4944-892b-8a683485db7e" containerName="limitador" containerID="cri-o://7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534" gracePeriod=30
Jul 07 18:32:24.949759 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:24.949737    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:25.041999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.041915    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/265f1c21-d4dc-4944-892b-8a683485db7e-config-file\") pod \"265f1c21-d4dc-4944-892b-8a683485db7e\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") "
Jul 07 18:32:25.042140 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.042019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqjrm\" (UniqueName: \"kubernetes.io/projected/265f1c21-d4dc-4944-892b-8a683485db7e-kube-api-access-fqjrm\") pod \"265f1c21-d4dc-4944-892b-8a683485db7e\" (UID: \"265f1c21-d4dc-4944-892b-8a683485db7e\") "
Jul 07 18:32:25.042309 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.042289    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265f1c21-d4dc-4944-892b-8a683485db7e-config-file" (OuterVolumeSpecName: "config-file") pod "265f1c21-d4dc-4944-892b-8a683485db7e" (UID: "265f1c21-d4dc-4944-892b-8a683485db7e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:32:25.044166 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.044133    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265f1c21-d4dc-4944-892b-8a683485db7e-kube-api-access-fqjrm" (OuterVolumeSpecName: "kube-api-access-fqjrm") pod "265f1c21-d4dc-4944-892b-8a683485db7e" (UID: "265f1c21-d4dc-4944-892b-8a683485db7e"). InnerVolumeSpecName "kube-api-access-fqjrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:32:25.143173 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.143122    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/265f1c21-d4dc-4944-892b-8a683485db7e-config-file\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:32:25.143173 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.143167    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjrm\" (UniqueName: \"kubernetes.io/projected/265f1c21-d4dc-4944-892b-8a683485db7e-kube-api-access-fqjrm\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:32:25.391435 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.391396    2565 generic.go:356] "Generic (PLEG): container finished" podID="265f1c21-d4dc-4944-892b-8a683485db7e" containerID="7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534" exitCode=0
Jul 07 18:32:25.391860 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.391477    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl"
Jul 07 18:32:25.391860 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.391481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl" event={"ID":"265f1c21-d4dc-4944-892b-8a683485db7e","Type":"ContainerDied","Data":"7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534"}
Jul 07 18:32:25.391860 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.391616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rckdl" event={"ID":"265f1c21-d4dc-4944-892b-8a683485db7e","Type":"ContainerDied","Data":"b789139fe5f1ce996f63b6a700d604c270d82b897b414e7360500b0f28994ffd"}
Jul 07 18:32:25.391860 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.391637    2565 scope.go:117] "RemoveContainer" containerID="7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534"
Jul 07 18:32:25.399486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.399471    2565 scope.go:117] "RemoveContainer" containerID="7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534"
Jul 07 18:32:25.399733 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:32:25.399706    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534\": container with ID starting with 7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534 not found: ID does not exist" containerID="7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534"
Jul 07 18:32:25.399787 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.399741    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534"} err="failed to get container status \"7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534\": rpc error: code = NotFound desc = could not find container \"7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534\": container with ID starting with 7a42982412025b849ff03925fb0bf6e6b9431ddb91b20f46a41f90a9a2f8b534 not found: ID does not exist"
Jul 07 18:32:25.412530 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.412507    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:25.416390 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:25.416371    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rckdl"]
Jul 07 18:32:27.048018 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:27.047984    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265f1c21-d4dc-4944-892b-8a683485db7e" path="/var/lib/kubelet/pods/265f1c21-d4dc-4944-892b-8a683485db7e/volumes"
Jul 07 18:32:47.520858 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:47.520822    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gtbnq"]
Jul 07 18:32:47.521263 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:47.521036    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gtbnq" podUID="6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" containerName="authorino" containerID="cri-o://3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a" gracePeriod=30
Jul 07 18:32:47.759613 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:47.759587    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:47.834654 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:47.834626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxhjc\" (UniqueName: \"kubernetes.io/projected/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7-kube-api-access-wxhjc\") pod \"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7\" (UID: \"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7\") "
Jul 07 18:32:47.836665 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:47.836636    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7-kube-api-access-wxhjc" (OuterVolumeSpecName: "kube-api-access-wxhjc") pod "6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" (UID: "6eaeaa5e-1feb-4851-aba9-7691ed7e64e7"). InnerVolumeSpecName "kube-api-access-wxhjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:32:47.935555 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:47.935527    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhjc\" (UniqueName: \"kubernetes.io/projected/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7-kube-api-access-wxhjc\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:32:48.493547 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.493513    2565 generic.go:356] "Generic (PLEG): container finished" podID="6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" containerID="3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a" exitCode=0
Jul 07 18:32:48.493718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.493556    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gtbnq"
Jul 07 18:32:48.493718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.493590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gtbnq" event={"ID":"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7","Type":"ContainerDied","Data":"3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a"}
Jul 07 18:32:48.493718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.493632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gtbnq" event={"ID":"6eaeaa5e-1feb-4851-aba9-7691ed7e64e7","Type":"ContainerDied","Data":"adc68df8b52674d2af96e98ff023b79411ca209cc642f9ea16017c1ec3b9fec2"}
Jul 07 18:32:48.493718 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.493654    2565 scope.go:117] "RemoveContainer" containerID="3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a"
Jul 07 18:32:48.501638 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.501615    2565 scope.go:117] "RemoveContainer" containerID="3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a"
Jul 07 18:32:48.501921 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:32:48.501897    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a\": container with ID starting with 3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a not found: ID does not exist" containerID="3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a"
Jul 07 18:32:48.502003 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.501928    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a"} err="failed to get container status \"3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a\": rpc error: code = NotFound desc = could not find container \"3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a\": container with ID starting with 3a21d98b680f6143018740ee51d989cec05d83eb51d2fd49282f3646e47d471a not found: ID does not exist"
Jul 07 18:32:48.515835 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.515809    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gtbnq"]
Jul 07 18:32:48.520247 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:48.520226    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gtbnq"]
Jul 07 18:32:49.048296 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:49.048255    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" path="/var/lib/kubelet/pods/6eaeaa5e-1feb-4851-aba9-7691ed7e64e7/volumes"
Jul 07 18:32:52.338051 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338014    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-657db75cfb-hsc2t"]
Jul 07 18:32:52.338497 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" containerName="authorino"
Jul 07 18:32:52.338558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338501    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" containerName="authorino"
Jul 07 18:32:52.338558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338526    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265f1c21-d4dc-4944-892b-8a683485db7e" containerName="limitador"
Jul 07 18:32:52.338558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338535    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="265f1c21-d4dc-4944-892b-8a683485db7e" containerName="limitador"
Jul 07 18:32:52.338654 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338599    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="265f1c21-d4dc-4944-892b-8a683485db7e" containerName="limitador"
Jul 07 18:32:52.338654 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.338616    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eaeaa5e-1feb-4851-aba9-7691ed7e64e7" containerName="authorino"
Jul 07 18:32:52.341638 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.341619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.344431 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.344408    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 18:32:52.344562 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.344454    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 18:32:52.344562 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.344454    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-twv66\""
Jul 07 18:32:52.352563 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.352535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-657db75cfb-hsc2t"]
Jul 07 18:32:52.480355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.480317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202bd88b-5946-4235-bdc9-023bc6a2a02a-webhook-cert\") pod \"maas-controller-657db75cfb-hsc2t\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") " pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.480355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.480356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blgn\" (UniqueName: \"kubernetes.io/projected/202bd88b-5946-4235-bdc9-023bc6a2a02a-kube-api-access-2blgn\") pod \"maas-controller-657db75cfb-hsc2t\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") " pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.541823 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.541793    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787c79d564-l7qct"]
Jul 07 18:32:52.544102 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.544083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.554594 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.554576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787c79d564-l7qct"]
Jul 07 18:32:52.581638 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.581607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202bd88b-5946-4235-bdc9-023bc6a2a02a-webhook-cert\") pod \"maas-controller-657db75cfb-hsc2t\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") " pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.581638 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.581639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blgn\" (UniqueName: \"kubernetes.io/projected/202bd88b-5946-4235-bdc9-023bc6a2a02a-kube-api-access-2blgn\") pod \"maas-controller-657db75cfb-hsc2t\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") " pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.583969 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.583946    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202bd88b-5946-4235-bdc9-023bc6a2a02a-webhook-cert\") pod \"maas-controller-657db75cfb-hsc2t\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") " pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.590584 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.590529    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blgn\" (UniqueName: \"kubernetes.io/projected/202bd88b-5946-4235-bdc9-023bc6a2a02a-kube-api-access-2blgn\") pod \"maas-controller-657db75cfb-hsc2t\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") " pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.652026 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.652001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:52.682148 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.682124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdxp\" (UniqueName: \"kubernetes.io/projected/1295486f-a977-4d69-858e-9010062bbba1-kube-api-access-fmdxp\") pod \"maas-controller-787c79d564-l7qct\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") " pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.682254 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.682184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1295486f-a977-4d69-858e-9010062bbba1-webhook-cert\") pod \"maas-controller-787c79d564-l7qct\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") " pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.784790 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.784709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-657db75cfb-hsc2t"]
Jul 07 18:32:52.784891 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.784834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdxp\" (UniqueName: \"kubernetes.io/projected/1295486f-a977-4d69-858e-9010062bbba1-kube-api-access-fmdxp\") pod \"maas-controller-787c79d564-l7qct\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") " pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.784984 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.784959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1295486f-a977-4d69-858e-9010062bbba1-webhook-cert\") pod \"maas-controller-787c79d564-l7qct\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") " pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.786706 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:32:52.786667    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202bd88b_5946_4235_bdc9_023bc6a2a02a.slice/crio-3754fa3e19c6c1b9db12df9dbb0854f72176b16a4ffdb13caaac70de311eb12a WatchSource:0}: Error finding container 3754fa3e19c6c1b9db12df9dbb0854f72176b16a4ffdb13caaac70de311eb12a: Status 404 returned error can't find the container with id 3754fa3e19c6c1b9db12df9dbb0854f72176b16a4ffdb13caaac70de311eb12a
Jul 07 18:32:52.787930 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.787903    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1295486f-a977-4d69-858e-9010062bbba1-webhook-cert\") pod \"maas-controller-787c79d564-l7qct\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") " pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.793875 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.793849    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdxp\" (UniqueName: \"kubernetes.io/projected/1295486f-a977-4d69-858e-9010062bbba1-kube-api-access-fmdxp\") pod \"maas-controller-787c79d564-l7qct\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") " pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.854652 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.854600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:52.985285 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:52.985261    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787c79d564-l7qct"]
Jul 07 18:32:52.987908 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:32:52.987885    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1295486f_a977_4d69_858e_9010062bbba1.slice/crio-6f81b1108f0240f23fbd58d5d9ba63809745d520cd4a5d0d5b8e918e07eb496a WatchSource:0}: Error finding container 6f81b1108f0240f23fbd58d5d9ba63809745d520cd4a5d0d5b8e918e07eb496a: Status 404 returned error can't find the container with id 6f81b1108f0240f23fbd58d5d9ba63809745d520cd4a5d0d5b8e918e07eb496a
Jul 07 18:32:53.520207 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:53.520165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787c79d564-l7qct" event={"ID":"1295486f-a977-4d69-858e-9010062bbba1","Type":"ContainerStarted","Data":"6f81b1108f0240f23fbd58d5d9ba63809745d520cd4a5d0d5b8e918e07eb496a"}
Jul 07 18:32:53.521824 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:53.521739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657db75cfb-hsc2t" event={"ID":"202bd88b-5946-4235-bdc9-023bc6a2a02a","Type":"ContainerStarted","Data":"3754fa3e19c6c1b9db12df9dbb0854f72176b16a4ffdb13caaac70de311eb12a"}
Jul 07 18:32:55.228804 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.228779    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 18:32:55.530716 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.530629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657db75cfb-hsc2t" event={"ID":"202bd88b-5946-4235-bdc9-023bc6a2a02a","Type":"ContainerStarted","Data":"3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6"}
Jul 07 18:32:55.530716 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.530710    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:32:55.532113 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.532083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787c79d564-l7qct" event={"ID":"1295486f-a977-4d69-858e-9010062bbba1","Type":"ContainerStarted","Data":"d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14"}
Jul 07 18:32:55.532233 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.532198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:32:55.554449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.554391    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-657db75cfb-hsc2t" podStartSLOduration=1.116726334 podStartE2EDuration="3.554377013s" totalImagesPullingTime="2.437650679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:55.552106623 +0000 UTC m=+603.127870535" watchObservedRunningTime="2026-07-07 18:32:55.554377013 +0000 UTC m=+603.130140908"
Jul 07 18:32:55.596366 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:32:55.596302    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787c79d564-l7qct" podStartSLOduration=1.351697376 podStartE2EDuration="3.596287192s" totalImagesPullingTime="2.244589816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:55.595436179 +0000 UTC m=+603.171200094" watchObservedRunningTime="2026-07-07 18:32:55.596287192 +0000 UTC m=+603.172051086"
Jul 07 18:33:06.542571 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:06.542539    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:33:06.542963 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:06.542610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:33:06.657696 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:06.657663    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-657db75cfb-hsc2t"]
Jul 07 18:33:06.657884 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:06.657847    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-657db75cfb-hsc2t" podUID="202bd88b-5946-4235-bdc9-023bc6a2a02a" containerName="manager" containerID="cri-o://3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6" gracePeriod=30
Jul 07 18:33:06.897802 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:06.897777    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:33:07.001149 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.001121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2blgn\" (UniqueName: \"kubernetes.io/projected/202bd88b-5946-4235-bdc9-023bc6a2a02a-kube-api-access-2blgn\") pod \"202bd88b-5946-4235-bdc9-023bc6a2a02a\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") "
Jul 07 18:33:07.001301 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.001183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202bd88b-5946-4235-bdc9-023bc6a2a02a-webhook-cert\") pod \"202bd88b-5946-4235-bdc9-023bc6a2a02a\" (UID: \"202bd88b-5946-4235-bdc9-023bc6a2a02a\") "
Jul 07 18:33:07.003348 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.003318    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202bd88b-5946-4235-bdc9-023bc6a2a02a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "202bd88b-5946-4235-bdc9-023bc6a2a02a" (UID: "202bd88b-5946-4235-bdc9-023bc6a2a02a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:33:07.003478 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.003327    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202bd88b-5946-4235-bdc9-023bc6a2a02a-kube-api-access-2blgn" (OuterVolumeSpecName: "kube-api-access-2blgn") pod "202bd88b-5946-4235-bdc9-023bc6a2a02a" (UID: "202bd88b-5946-4235-bdc9-023bc6a2a02a"). InnerVolumeSpecName "kube-api-access-2blgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:33:07.102491 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.102407    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202bd88b-5946-4235-bdc9-023bc6a2a02a-webhook-cert\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:33:07.102491 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.102433    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blgn\" (UniqueName: \"kubernetes.io/projected/202bd88b-5946-4235-bdc9-023bc6a2a02a-kube-api-access-2blgn\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:33:07.587424 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.587385    2565 generic.go:356] "Generic (PLEG): container finished" podID="202bd88b-5946-4235-bdc9-023bc6a2a02a" containerID="3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6" exitCode=0
Jul 07 18:33:07.587852 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.587433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657db75cfb-hsc2t" event={"ID":"202bd88b-5946-4235-bdc9-023bc6a2a02a","Type":"ContainerDied","Data":"3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6"}
Jul 07 18:33:07.587852 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.587444    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657db75cfb-hsc2t"
Jul 07 18:33:07.587852 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.587527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657db75cfb-hsc2t" event={"ID":"202bd88b-5946-4235-bdc9-023bc6a2a02a","Type":"ContainerDied","Data":"3754fa3e19c6c1b9db12df9dbb0854f72176b16a4ffdb13caaac70de311eb12a"}
Jul 07 18:33:07.587852 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.587559    2565 scope.go:117] "RemoveContainer" containerID="3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6"
Jul 07 18:33:07.595576 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.595557    2565 scope.go:117] "RemoveContainer" containerID="3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6"
Jul 07 18:33:07.595833 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:33:07.595811    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6\": container with ID starting with 3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6 not found: ID does not exist" containerID="3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6"
Jul 07 18:33:07.595907 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.595838    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6"} err="failed to get container status \"3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6\": rpc error: code = NotFound desc = could not find container \"3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6\": container with ID starting with 3eb65a9828e54a3d4285d33dbaf193dad98b193c99c5955ab18c9b9ffb936ef6 not found: ID does not exist"
Jul 07 18:33:07.604051 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.604029    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-657db75cfb-hsc2t"]
Jul 07 18:33:07.607925 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:07.607904    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-657db75cfb-hsc2t"]
Jul 07 18:33:09.047451 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:09.047422    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="202bd88b-5946-4235-bdc9-023bc6a2a02a" path="/var/lib/kubelet/pods/202bd88b-5946-4235-bdc9-023bc6a2a02a/volumes"
Jul 07 18:33:28.597562 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.597446    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5c6877d755-667k6"]
Jul 07 18:33:28.598130 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.597791    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202bd88b-5946-4235-bdc9-023bc6a2a02a" containerName="manager"
Jul 07 18:33:28.598130 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.597803    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="202bd88b-5946-4235-bdc9-023bc6a2a02a" containerName="manager"
Jul 07 18:33:28.598130 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.597865    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="202bd88b-5946-4235-bdc9-023bc6a2a02a" containerName="manager"
Jul 07 18:33:28.611975 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.611945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.614388 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.614359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5c6877d755-667k6"]
Jul 07 18:33:28.615404 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.615384    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 07 18:33:28.673924 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.673886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfk8v\" (UniqueName: \"kubernetes.io/projected/a5f32407-ebfc-426f-b950-97b53af468c3-kube-api-access-gfk8v\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.674098 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.673945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f32407-ebfc-426f-b950-97b53af468c3-openshift-service-ca\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.674098 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.674015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f32407-ebfc-426f-b950-97b53af468c3-maas-api-tls\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.774883 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.774844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f32407-ebfc-426f-b950-97b53af468c3-maas-api-tls\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.775064 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.774915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfk8v\" (UniqueName: \"kubernetes.io/projected/a5f32407-ebfc-426f-b950-97b53af468c3-kube-api-access-gfk8v\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.775064 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.774977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f32407-ebfc-426f-b950-97b53af468c3-openshift-service-ca\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.775746 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.775723    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f32407-ebfc-426f-b950-97b53af468c3-openshift-service-ca\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.777432 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.777407    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f32407-ebfc-426f-b950-97b53af468c3-maas-api-tls\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.786372 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.786345    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfk8v\" (UniqueName: \"kubernetes.io/projected/a5f32407-ebfc-426f-b950-97b53af468c3-kube-api-access-gfk8v\") pod \"maas-api-5c6877d755-667k6\" (UID: \"a5f32407-ebfc-426f-b950-97b53af468c3\") " pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:28.924387 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:28.924303    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:29.069814 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.069791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5c6877d755-667k6"]
Jul 07 18:33:29.071767 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:33:29.071743    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f32407_ebfc_426f_b950_97b53af468c3.slice/crio-2dc6f40cf753c8bb9408d76902a1593bdf414c2c845739145e0a4e9bb15f4097 WatchSource:0}: Error finding container 2dc6f40cf753c8bb9408d76902a1593bdf414c2c845739145e0a4e9bb15f4097: Status 404 returned error can't find the container with id 2dc6f40cf753c8bb9408d76902a1593bdf414c2c845739145e0a4e9bb15f4097
Jul 07 18:33:29.543963 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.543926    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-z5fsx"]
Jul 07 18:33:29.548911 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.548878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.554293 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.554268    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 18:33:29.554826 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.554810    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5q9pl\""
Jul 07 18:33:29.564788 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.564756    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-z5fsx"]
Jul 07 18:33:29.581996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.581949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhs2\" (UniqueName: \"kubernetes.io/projected/2dfa39d3-330f-41aa-85d1-a9abb00ad516-kube-api-access-jbhs2\") pod \"payload-processing-58f86c7798-z5fsx\" (UID: \"2dfa39d3-330f-41aa-85d1-a9abb00ad516\") " pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.581996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.581988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfa39d3-330f-41aa-85d1-a9abb00ad516-plugins-config-volume\") pod \"payload-processing-58f86c7798-z5fsx\" (UID: \"2dfa39d3-330f-41aa-85d1-a9abb00ad516\") " pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.682510 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.682386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhs2\" (UniqueName: \"kubernetes.io/projected/2dfa39d3-330f-41aa-85d1-a9abb00ad516-kube-api-access-jbhs2\") pod \"payload-processing-58f86c7798-z5fsx\" (UID: \"2dfa39d3-330f-41aa-85d1-a9abb00ad516\") " pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.682973 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.682520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfa39d3-330f-41aa-85d1-a9abb00ad516-plugins-config-volume\") pod \"payload-processing-58f86c7798-z5fsx\" (UID: \"2dfa39d3-330f-41aa-85d1-a9abb00ad516\") " pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.683217 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.683195    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfa39d3-330f-41aa-85d1-a9abb00ad516-plugins-config-volume\") pod \"payload-processing-58f86c7798-z5fsx\" (UID: \"2dfa39d3-330f-41aa-85d1-a9abb00ad516\") " pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.686664 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.686638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5c6877d755-667k6" event={"ID":"a5f32407-ebfc-426f-b950-97b53af468c3","Type":"ContainerStarted","Data":"2dc6f40cf753c8bb9408d76902a1593bdf414c2c845739145e0a4e9bb15f4097"}
Jul 07 18:33:29.702179 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.702148    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhs2\" (UniqueName: \"kubernetes.io/projected/2dfa39d3-330f-41aa-85d1-a9abb00ad516-kube-api-access-jbhs2\") pod \"payload-processing-58f86c7798-z5fsx\" (UID: \"2dfa39d3-330f-41aa-85d1-a9abb00ad516\") " pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:29.860707 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:29.860666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:30.008968 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:30.008941    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-z5fsx"]
Jul 07 18:33:30.011451 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:33:30.011414    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfa39d3_330f_41aa_85d1_a9abb00ad516.slice/crio-677b1c6febedbbfb7ff4435ff83a8ae3151be72f854670411b7bc4e05cbfa93d WatchSource:0}: Error finding container 677b1c6febedbbfb7ff4435ff83a8ae3151be72f854670411b7bc4e05cbfa93d: Status 404 returned error can't find the container with id 677b1c6febedbbfb7ff4435ff83a8ae3151be72f854670411b7bc4e05cbfa93d
Jul 07 18:33:30.693222 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:30.693145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx" event={"ID":"2dfa39d3-330f-41aa-85d1-a9abb00ad516","Type":"ContainerStarted","Data":"677b1c6febedbbfb7ff4435ff83a8ae3151be72f854670411b7bc4e05cbfa93d"}
Jul 07 18:33:31.700903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:31.700864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx" event={"ID":"2dfa39d3-330f-41aa-85d1-a9abb00ad516","Type":"ContainerStarted","Data":"92a04cb9aa5a5666e5d0a5b4b759342d551761e83a93d692cc1860cce4532b69"}
Jul 07 18:33:31.701558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:31.700984    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:31.702825 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:31.702800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5c6877d755-667k6" event={"ID":"a5f32407-ebfc-426f-b950-97b53af468c3","Type":"ContainerStarted","Data":"9d6e5ef0147f041e007d439483ff1c946a9030b2ff30c321065438165fffe665"}
Jul 07 18:33:31.702966 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:31.702943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:31.730014 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:31.729961    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx" podStartSLOduration=1.225401648 podStartE2EDuration="2.729942409s" totalImagesPullingTime="1.504540761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:33:31.728685621 +0000 UTC m=+639.304449516" watchObservedRunningTime="2026-07-07 18:33:31.729942409 +0000 UTC m=+639.305706283"
Jul 07 18:33:31.748194 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:31.748141    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5c6877d755-667k6" podStartSLOduration=1.872878703 podStartE2EDuration="3.748124118s" totalImagesPullingTime="1.875245415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:33:31.746835887 +0000 UTC m=+639.322599847" watchObservedRunningTime="2026-07-07 18:33:31.748124118 +0000 UTC m=+639.323888011"
Jul 07 18:33:37.713558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:37.713525    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5c6877d755-667k6"
Jul 07 18:33:42.710230 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:42.710195    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-z5fsx"
Jul 07 18:33:55.241987 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.241957    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"]
Jul 07 18:33:55.249231 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.249206    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.253334 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.253309    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 18:33:55.253334 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.253329    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 18:33:55.254399 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.254381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8spls\""
Jul 07 18:33:55.254399 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.254404    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 07 18:33:55.259255 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.259233    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"]
Jul 07 18:33:55.393197 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.393155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.393363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.393205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f17f0-538a-4884-bde5-4bd83c0b3928-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.393363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.393238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.393363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.393259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.393363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.393282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkhv\" (UniqueName: \"kubernetes.io/projected/659f17f0-538a-4884-bde5-4bd83c0b3928-kube-api-access-qzkhv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.393363 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.393351    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.494558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.494558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkhv\" (UniqueName: \"kubernetes.io/projected/659f17f0-538a-4884-bde5-4bd83c0b3928-kube-api-access-qzkhv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.494558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.494789 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.494789 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f17f0-538a-4884-bde5-4bd83c0b3928-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.494789 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.495025 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.494995    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.495025 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.495010    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.495163 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.495065    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.496875 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.496851    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f17f0-538a-4884-bde5-4bd83c0b3928-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.497180 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.497158    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f17f0-538a-4884-bde5-4bd83c0b3928-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.502358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.502334    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkhv\" (UniqueName: \"kubernetes.io/projected/659f17f0-538a-4884-bde5-4bd83c0b3928-kube-api-access-qzkhv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg\" (UID: \"659f17f0-538a-4884-bde5-4bd83c0b3928\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.560397 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.560375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:33:55.689169 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.689144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"]
Jul 07 18:33:55.691749 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:33:55.691714    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659f17f0_538a_4884_bde5_4bd83c0b3928.slice/crio-acdb2cdddef45fabe7e2734189fe97ed68c7d88aef1d5c39d7c4f803b54bb30d WatchSource:0}: Error finding container acdb2cdddef45fabe7e2734189fe97ed68c7d88aef1d5c39d7c4f803b54bb30d: Status 404 returned error can't find the container with id acdb2cdddef45fabe7e2734189fe97ed68c7d88aef1d5c39d7c4f803b54bb30d
Jul 07 18:33:55.693203 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.693187    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:33:55.805956 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:55.805875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg" event={"ID":"659f17f0-538a-4884-bde5-4bd83c0b3928","Type":"ContainerStarted","Data":"acdb2cdddef45fabe7e2734189fe97ed68c7d88aef1d5c39d7c4f803b54bb30d"}
Jul 07 18:33:56.812150 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:56.812119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg" event={"ID":"659f17f0-538a-4884-bde5-4bd83c0b3928","Type":"ContainerStarted","Data":"1b5b7ad8e03d4005585606b65d2f67918e8848049c1f1fa3a0d3e80502759ea1"}
Jul 07 18:33:56.830575 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:33:56.830529    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg" podStartSLOduration=0.807115698 podStartE2EDuration="1.830514412s" totalImagesPullingTime="1.023398714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:33:56.829370314 +0000 UTC m=+664.405134209" watchObservedRunningTime="2026-07-07 18:33:56.830514412 +0000 UTC m=+664.406278305"
Jul 07 18:34:05.561384 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:05.561355    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:34:05.561919 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:05.561426    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:34:05.573870 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:05.573841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:34:05.861603 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:05.861578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg"
Jul 07 18:34:07.330290 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.330256    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"]
Jul 07 18:34:07.336398 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.336377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.339161 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.339142    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 07 18:34:07.347035 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.347012    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"]
Jul 07 18:34:07.391867 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.391837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.391985 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.391885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.391985 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.391945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602a30e3-77c9-4efb-85d3-b8d5141789ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.392056 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.392004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjr29\" (UniqueName: \"kubernetes.io/projected/602a30e3-77c9-4efb-85d3-b8d5141789ce-kube-api-access-gjr29\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.392056 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.392027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.392056 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.392048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492371 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492507 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492507 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602a30e3-77c9-4efb-85d3-b8d5141789ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjr29\" (UniqueName: \"kubernetes.io/projected/602a30e3-77c9-4efb-85d3-b8d5141789ce-kube-api-access-gjr29\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492936 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492906    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.492936 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492933    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.493144 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.492962    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.494896 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.494873    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602a30e3-77c9-4efb-85d3-b8d5141789ce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.495027 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.495008    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602a30e3-77c9-4efb-85d3-b8d5141789ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.501032 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.501009    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjr29\" (UniqueName: \"kubernetes.io/projected/602a30e3-77c9-4efb-85d3-b8d5141789ce-kube-api-access-gjr29\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5xtmq\" (UID: \"602a30e3-77c9-4efb-85d3-b8d5141789ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.646811 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.646728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:07.788613 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.788591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"]
Jul 07 18:34:07.790864 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:34:07.790830    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602a30e3_77c9_4efb_85d3_b8d5141789ce.slice/crio-c043e81b05674540f240baf56e4f20dc9d577625e5792efc2c4c2c4096e9ec28 WatchSource:0}: Error finding container c043e81b05674540f240baf56e4f20dc9d577625e5792efc2c4c2c4096e9ec28: Status 404 returned error can't find the container with id c043e81b05674540f240baf56e4f20dc9d577625e5792efc2c4c2c4096e9ec28
Jul 07 18:34:07.860260 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:07.860233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq" event={"ID":"602a30e3-77c9-4efb-85d3-b8d5141789ce","Type":"ContainerStarted","Data":"c043e81b05674540f240baf56e4f20dc9d577625e5792efc2c4c2c4096e9ec28"}
Jul 07 18:34:08.866574 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:08.866533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq" event={"ID":"602a30e3-77c9-4efb-85d3-b8d5141789ce","Type":"ContainerStarted","Data":"81b13d0482f7b27d7c0b06fa3cb66017a8405233a9712e7a071c1ad3e0de6884"}
Jul 07 18:34:08.886921 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:08.886866    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq" podStartSLOduration=1.6452961529999999 podStartE2EDuration="1.886850099s" totalImagesPullingTime="241.553946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:34:08.885592825 +0000 UTC m=+676.461356720" watchObservedRunningTime="2026-07-07 18:34:08.886850099 +0000 UTC m=+676.462614044"
Jul 07 18:34:17.647849 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:17.647813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:17.648254 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:17.647891    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:17.660650 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:17.660627    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:17.918218 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:17.918133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5xtmq"
Jul 07 18:34:19.434765 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.434732    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"]
Jul 07 18:34:19.438146 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.438126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.442402 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.442371    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 07 18:34:19.487912 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.487886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5ae3a1-302e-4d61-94bb-1d090abf2896-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.488025 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.487934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.488025 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.487956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.488101 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.488033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.488101 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.488072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vrn\" (UniqueName: \"kubernetes.io/projected/ea5ae3a1-302e-4d61-94bb-1d090abf2896-kube-api-access-b6vrn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.488164 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.488100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.496578 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.496550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"]
Jul 07 18:34:19.589179 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vrn\" (UniqueName: \"kubernetes.io/projected/ea5ae3a1-302e-4d61-94bb-1d090abf2896-kube-api-access-b6vrn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589179 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589379 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5ae3a1-302e-4d61-94bb-1d090abf2896-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589379 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589379 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589509 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589637 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589616    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589801 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589773    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.589801 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.589796    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.591634 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.591604    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5ae3a1-302e-4d61-94bb-1d090abf2896-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.591803 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.591788    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5ae3a1-302e-4d61-94bb-1d090abf2896-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.602679 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.602661    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vrn\" (UniqueName: \"kubernetes.io/projected/ea5ae3a1-302e-4d61-94bb-1d090abf2896-kube-api-access-b6vrn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs\" (UID: \"ea5ae3a1-302e-4d61-94bb-1d090abf2896\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.749284 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.749177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:19.881513 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.881487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"]
Jul 07 18:34:19.883424 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:34:19.883397    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5ae3a1_302e_4d61_94bb_1d090abf2896.slice/crio-724bbd0a84bcdcc5d2001b22748acfa59ced5bd41a31a9443c35d7a45f59dfb7 WatchSource:0}: Error finding container 724bbd0a84bcdcc5d2001b22748acfa59ced5bd41a31a9443c35d7a45f59dfb7: Status 404 returned error can't find the container with id 724bbd0a84bcdcc5d2001b22748acfa59ced5bd41a31a9443c35d7a45f59dfb7
Jul 07 18:34:19.916516 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:19.916489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs" event={"ID":"ea5ae3a1-302e-4d61-94bb-1d090abf2896","Type":"ContainerStarted","Data":"724bbd0a84bcdcc5d2001b22748acfa59ced5bd41a31a9443c35d7a45f59dfb7"}
Jul 07 18:34:20.921919 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:20.921885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs" event={"ID":"ea5ae3a1-302e-4d61-94bb-1d090abf2896","Type":"ContainerStarted","Data":"bb615963e4fc336c6bf8317d1a33723f1acb2ebaf152f9ecaf942d9ecce52ca5"}
Jul 07 18:34:20.944436 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:20.944385    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs" podStartSLOduration=1.737212332 podStartE2EDuration="1.944368348s" totalImagesPullingTime="207.156016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:34:20.941846269 +0000 UTC m=+688.517610162" watchObservedRunningTime="2026-07-07 18:34:20.944368348 +0000 UTC m=+688.520132239"
Jul 07 18:34:29.749710 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:29.749678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:29.750146 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:29.749728    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:29.762067 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:29.762043    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:34:29.976165 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:34:29.976141    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs"
Jul 07 18:36:14.257648 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.257614    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-789b6bd8c4-v8v98"]
Jul 07 18:36:14.261004 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.260988    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.267668 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.267641    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p868l\""
Jul 07 18:36:14.267796 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.267704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 18:36:14.282283 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.282260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-789b6bd8c4-v8v98"]
Jul 07 18:36:14.329737 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.329707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwsb\" (UniqueName: \"kubernetes.io/projected/3d9a5aa9-c9cc-4480-851c-d180c39e5972-kube-api-access-wkwsb\") pod \"authorino-789b6bd8c4-v8v98\" (UID: \"3d9a5aa9-c9cc-4480-851c-d180c39e5972\") " pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.329869 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.329832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d9a5aa9-c9cc-4480-851c-d180c39e5972-tls-cert\") pod \"authorino-789b6bd8c4-v8v98\" (UID: \"3d9a5aa9-c9cc-4480-851c-d180c39e5972\") " pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.431075 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.431045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d9a5aa9-c9cc-4480-851c-d180c39e5972-tls-cert\") pod \"authorino-789b6bd8c4-v8v98\" (UID: \"3d9a5aa9-c9cc-4480-851c-d180c39e5972\") " pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.431243 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.431105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwsb\" (UniqueName: \"kubernetes.io/projected/3d9a5aa9-c9cc-4480-851c-d180c39e5972-kube-api-access-wkwsb\") pod \"authorino-789b6bd8c4-v8v98\" (UID: \"3d9a5aa9-c9cc-4480-851c-d180c39e5972\") " pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.433756 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.433731    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d9a5aa9-c9cc-4480-851c-d180c39e5972-tls-cert\") pod \"authorino-789b6bd8c4-v8v98\" (UID: \"3d9a5aa9-c9cc-4480-851c-d180c39e5972\") " pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.456086 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.456058    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwsb\" (UniqueName: \"kubernetes.io/projected/3d9a5aa9-c9cc-4480-851c-d180c39e5972-kube-api-access-wkwsb\") pod \"authorino-789b6bd8c4-v8v98\" (UID: \"3d9a5aa9-c9cc-4480-851c-d180c39e5972\") " pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.569754 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.569663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-789b6bd8c4-v8v98"
Jul 07 18:36:14.717041 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:14.717014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-789b6bd8c4-v8v98"]
Jul 07 18:36:14.718357 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:36:14.718327    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d9a5aa9_c9cc_4480_851c_d180c39e5972.slice/crio-4faad42702bf47e580e1aa19781d6122f678e31f58f26da263987a60206ebbb5 WatchSource:0}: Error finding container 4faad42702bf47e580e1aa19781d6122f678e31f58f26da263987a60206ebbb5: Status 404 returned error can't find the container with id 4faad42702bf47e580e1aa19781d6122f678e31f58f26da263987a60206ebbb5
Jul 07 18:36:15.440739 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:15.440706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-789b6bd8c4-v8v98" event={"ID":"3d9a5aa9-c9cc-4480-851c-d180c39e5972","Type":"ContainerStarted","Data":"34b92a9c938d38718a30086b696b47947bd933ffbd5317fee689f9e725297bea"}
Jul 07 18:36:15.441120 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:15.440754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-789b6bd8c4-v8v98" event={"ID":"3d9a5aa9-c9cc-4480-851c-d180c39e5972","Type":"ContainerStarted","Data":"4faad42702bf47e580e1aa19781d6122f678e31f58f26da263987a60206ebbb5"}
Jul 07 18:36:15.483750 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:36:15.483679    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-789b6bd8c4-v8v98" podStartSLOduration=1.133529874 podStartE2EDuration="1.483661867s" totalImagesPullingTime="350.131993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:36:15.482381868 +0000 UTC m=+803.058145780" watchObservedRunningTime="2026-07-07 18:36:15.483661867 +0000 UTC m=+803.059425738"
Jul 07 18:37:17.537511 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:17.537442    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-787c79d564-l7qct" podUID="1295486f-a977-4d69-858e-9010062bbba1" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.30:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 07 18:37:28.241725 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.241690    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787c79d564-l7qct"]
Jul 07 18:37:28.242117 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.241887    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-787c79d564-l7qct" podUID="1295486f-a977-4d69-858e-9010062bbba1" containerName="manager" containerID="cri-o://d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14" gracePeriod=30
Jul 07 18:37:28.488104 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.488082    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:37:28.538579 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.538518    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmdxp\" (UniqueName: \"kubernetes.io/projected/1295486f-a977-4d69-858e-9010062bbba1-kube-api-access-fmdxp\") pod \"1295486f-a977-4d69-858e-9010062bbba1\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") "
Jul 07 18:37:28.538579 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.538564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1295486f-a977-4d69-858e-9010062bbba1-webhook-cert\") pod \"1295486f-a977-4d69-858e-9010062bbba1\" (UID: \"1295486f-a977-4d69-858e-9010062bbba1\") "
Jul 07 18:37:28.540484 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.540434    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1295486f-a977-4d69-858e-9010062bbba1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1295486f-a977-4d69-858e-9010062bbba1" (UID: "1295486f-a977-4d69-858e-9010062bbba1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:37:28.540484 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.540452    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1295486f-a977-4d69-858e-9010062bbba1-kube-api-access-fmdxp" (OuterVolumeSpecName: "kube-api-access-fmdxp") pod "1295486f-a977-4d69-858e-9010062bbba1" (UID: "1295486f-a977-4d69-858e-9010062bbba1"). InnerVolumeSpecName "kube-api-access-fmdxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:37:28.639577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.639554    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdxp\" (UniqueName: \"kubernetes.io/projected/1295486f-a977-4d69-858e-9010062bbba1-kube-api-access-fmdxp\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:37:28.639577 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.639575    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1295486f-a977-4d69-858e-9010062bbba1-webhook-cert\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:37:28.769169 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.769142    2565 generic.go:356] "Generic (PLEG): container finished" podID="1295486f-a977-4d69-858e-9010062bbba1" containerID="d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14" exitCode=0
Jul 07 18:37:28.769287 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.769180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787c79d564-l7qct" event={"ID":"1295486f-a977-4d69-858e-9010062bbba1","Type":"ContainerDied","Data":"d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14"}
Jul 07 18:37:28.769287 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.769196    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787c79d564-l7qct"
Jul 07 18:37:28.769287 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.769232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787c79d564-l7qct" event={"ID":"1295486f-a977-4d69-858e-9010062bbba1","Type":"ContainerDied","Data":"6f81b1108f0240f23fbd58d5d9ba63809745d520cd4a5d0d5b8e918e07eb496a"}
Jul 07 18:37:28.769287 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.769255    2565 scope.go:117] "RemoveContainer" containerID="d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14"
Jul 07 18:37:28.778449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.778429    2565 scope.go:117] "RemoveContainer" containerID="d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14"
Jul 07 18:37:28.778757 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:37:28.778738    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14\": container with ID starting with d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14 not found: ID does not exist" containerID="d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14"
Jul 07 18:37:28.778798 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:28.778766    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14"} err="failed to get container status \"d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14\": rpc error: code = NotFound desc = could not find container \"d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14\": container with ID starting with d55085a9d367ce5c5263c0fa1141c83e362e1940a2fa267f42f866de3a019e14 not found: ID does not exist"
Jul 07 18:37:32.926571 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:32.926533    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787c79d564-l7qct"]
Jul 07 18:37:32.933525 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:32.933501    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-787c79d564-l7qct"]
Jul 07 18:37:33.046882 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:33.046849    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1295486f-a977-4d69-858e-9010062bbba1" path="/var/lib/kubelet/pods/1295486f-a977-4d69-858e-9010062bbba1/volumes"
Jul 07 18:37:34.096522 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.096488    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787c79d564-7vvgd"]
Jul 07 18:37:34.096892 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.096852    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1295486f-a977-4d69-858e-9010062bbba1" containerName="manager"
Jul 07 18:37:34.096892 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.096865    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1295486f-a977-4d69-858e-9010062bbba1" containerName="manager"
Jul 07 18:37:34.096978 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.096954    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1295486f-a977-4d69-858e-9010062bbba1" containerName="manager"
Jul 07 18:37:34.100904 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.100886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.107497 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.107452    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 18:37:34.107679 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.107661    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-twv66\""
Jul 07 18:37:34.107763 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.107696    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 18:37:34.121201 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.121181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787c79d564-7vvgd"]
Jul 07 18:37:34.185725 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.184751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf8c1666-ca1e-454e-9ba7-1cebfc225725-webhook-cert\") pod \"maas-controller-787c79d564-7vvgd\" (UID: \"cf8c1666-ca1e-454e-9ba7-1cebfc225725\") " pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.185725 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.184799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk69d\" (UniqueName: \"kubernetes.io/projected/cf8c1666-ca1e-454e-9ba7-1cebfc225725-kube-api-access-lk69d\") pod \"maas-controller-787c79d564-7vvgd\" (UID: \"cf8c1666-ca1e-454e-9ba7-1cebfc225725\") " pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.285786 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.285763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf8c1666-ca1e-454e-9ba7-1cebfc225725-webhook-cert\") pod \"maas-controller-787c79d564-7vvgd\" (UID: \"cf8c1666-ca1e-454e-9ba7-1cebfc225725\") " pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.285903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.285794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk69d\" (UniqueName: \"kubernetes.io/projected/cf8c1666-ca1e-454e-9ba7-1cebfc225725-kube-api-access-lk69d\") pod \"maas-controller-787c79d564-7vvgd\" (UID: \"cf8c1666-ca1e-454e-9ba7-1cebfc225725\") " pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.288127 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.288105    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf8c1666-ca1e-454e-9ba7-1cebfc225725-webhook-cert\") pod \"maas-controller-787c79d564-7vvgd\" (UID: \"cf8c1666-ca1e-454e-9ba7-1cebfc225725\") " pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.298912 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.298894    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk69d\" (UniqueName: \"kubernetes.io/projected/cf8c1666-ca1e-454e-9ba7-1cebfc225725-kube-api-access-lk69d\") pod \"maas-controller-787c79d564-7vvgd\" (UID: \"cf8c1666-ca1e-454e-9ba7-1cebfc225725\") " pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.411662 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.411608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:34.548790 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:37:34.548748    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8c1666_ca1e_454e_9ba7_1cebfc225725.slice/crio-0d7a4d2061672eb43ca947aa9d5a3b3bfc82cd8aa6b67d1cd4c3f9cc342c0c60 WatchSource:0}: Error finding container 0d7a4d2061672eb43ca947aa9d5a3b3bfc82cd8aa6b67d1cd4c3f9cc342c0c60: Status 404 returned error can't find the container with id 0d7a4d2061672eb43ca947aa9d5a3b3bfc82cd8aa6b67d1cd4c3f9cc342c0c60
Jul 07 18:37:34.553167 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.553140    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787c79d564-7vvgd"]
Jul 07 18:37:34.797873 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:34.797810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787c79d564-7vvgd" event={"ID":"cf8c1666-ca1e-454e-9ba7-1cebfc225725","Type":"ContainerStarted","Data":"0d7a4d2061672eb43ca947aa9d5a3b3bfc82cd8aa6b67d1cd4c3f9cc342c0c60"}
Jul 07 18:37:35.803829 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:35.803796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787c79d564-7vvgd" event={"ID":"cf8c1666-ca1e-454e-9ba7-1cebfc225725","Type":"ContainerStarted","Data":"0b8dc5baf3858e6afe590e1d2a36b0ce9dba0715129f4a4acacc7d9d88f33fdb"}
Jul 07 18:37:35.804226 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:35.803900    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:37:35.835446 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:35.835402    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787c79d564-7vvgd" podStartSLOduration=2.465437546 podStartE2EDuration="2.835389137s" totalImagesPullingTime="369.951591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:37:35.834561111 +0000 UTC m=+883.410325027" watchObservedRunningTime="2026-07-07 18:37:35.835389137 +0000 UTC m=+883.411153074"
Jul 07 18:37:46.814243 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:37:46.814168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787c79d564-7vvgd"
Jul 07 18:49:54.039678 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.039646    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"]
Jul 07 18:49:54.042954 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.042920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.045753 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.045730    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-4110137c-serving-cert\""
Jul 07 18:49:54.045876 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.045804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:49:54.061156 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.061133    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"]
Jul 07 18:49:54.091593 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.091571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczxb\" (UniqueName: \"kubernetes.io/projected/7e76e35f-13d3-4bbb-a4af-c83dba39856f-kube-api-access-gczxb\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.091710 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.091659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e76e35f-13d3-4bbb-a4af-c83dba39856f-maas-api-tls\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.091854 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.091836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e76e35f-13d3-4bbb-a4af-c83dba39856f-openshift-service-ca\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.192567 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.192545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczxb\" (UniqueName: \"kubernetes.io/projected/7e76e35f-13d3-4bbb-a4af-c83dba39856f-kube-api-access-gczxb\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.192685 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.192594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e76e35f-13d3-4bbb-a4af-c83dba39856f-maas-api-tls\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.192685 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.192636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e76e35f-13d3-4bbb-a4af-c83dba39856f-openshift-service-ca\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.193252 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.193232    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e76e35f-13d3-4bbb-a4af-c83dba39856f-openshift-service-ca\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.194974 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.194954    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e76e35f-13d3-4bbb-a4af-c83dba39856f-maas-api-tls\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.210346 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.210327    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczxb\" (UniqueName: \"kubernetes.io/projected/7e76e35f-13d3-4bbb-a4af-c83dba39856f-kube-api-access-gczxb\") pod \"maas-api-e2e-ait-4110137c-865784f88b-2zmzt\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") " pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.354646 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.354622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:54.504705 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.504682    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"]
Jul 07 18:49:54.506587 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:49:54.506561    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e76e35f_13d3_4bbb_a4af_c83dba39856f.slice/crio-6141863a4bfefd1cf33d98a14b82c3edc674cfd3770bf3974fd8f2df2c161af9 WatchSource:0}: Error finding container 6141863a4bfefd1cf33d98a14b82c3edc674cfd3770bf3974fd8f2df2c161af9: Status 404 returned error can't find the container with id 6141863a4bfefd1cf33d98a14b82c3edc674cfd3770bf3974fd8f2df2c161af9
Jul 07 18:49:54.507957 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:54.507912    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:49:55.085595 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:55.085563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt" event={"ID":"7e76e35f-13d3-4bbb-a4af-c83dba39856f","Type":"ContainerStarted","Data":"6141863a4bfefd1cf33d98a14b82c3edc674cfd3770bf3974fd8f2df2c161af9"}
Jul 07 18:49:56.092155 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:56.092121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt" event={"ID":"7e76e35f-13d3-4bbb-a4af-c83dba39856f","Type":"ContainerStarted","Data":"c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e"}
Jul 07 18:49:56.092600 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:56.092189    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:56.129853 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:56.129811    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt" podStartSLOduration=1.399069355 podStartE2EDuration="2.129797075s" totalImagesPullingTime="730.72772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:49:56.127112929 +0000 UTC m=+1623.702876822" watchObservedRunningTime="2026-07-07 18:49:56.129797075 +0000 UTC m=+1623.705560968"
Jul 07 18:49:59.578880 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.578850    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"]
Jul 07 18:49:59.579340 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.579014    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt" podUID="7e76e35f-13d3-4bbb-a4af-c83dba39856f" containerName="maas-api" containerID="cri-o://c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e" gracePeriod=30
Jul 07 18:49:59.824027 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.824003    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:49:59.937764 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.937691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e76e35f-13d3-4bbb-a4af-c83dba39856f-openshift-service-ca\") pod \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") "
Jul 07 18:49:59.937764 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.937738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gczxb\" (UniqueName: \"kubernetes.io/projected/7e76e35f-13d3-4bbb-a4af-c83dba39856f-kube-api-access-gczxb\") pod \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") "
Jul 07 18:49:59.937938 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.937782    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e76e35f-13d3-4bbb-a4af-c83dba39856f-maas-api-tls\") pod \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\" (UID: \"7e76e35f-13d3-4bbb-a4af-c83dba39856f\") "
Jul 07 18:49:59.938068 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.938042    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e76e35f-13d3-4bbb-a4af-c83dba39856f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e76e35f-13d3-4bbb-a4af-c83dba39856f" (UID: "7e76e35f-13d3-4bbb-a4af-c83dba39856f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:49:59.939827 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.939806    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e76e35f-13d3-4bbb-a4af-c83dba39856f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e76e35f-13d3-4bbb-a4af-c83dba39856f" (UID: "7e76e35f-13d3-4bbb-a4af-c83dba39856f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:49:59.939901 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:49:59.939806    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e76e35f-13d3-4bbb-a4af-c83dba39856f-kube-api-access-gczxb" (OuterVolumeSpecName: "kube-api-access-gczxb") pod "7e76e35f-13d3-4bbb-a4af-c83dba39856f" (UID: "7e76e35f-13d3-4bbb-a4af-c83dba39856f"). InnerVolumeSpecName "kube-api-access-gczxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:50:00.038607 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.038584    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e76e35f-13d3-4bbb-a4af-c83dba39856f-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:00.038607 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.038604    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczxb\" (UniqueName: \"kubernetes.io/projected/7e76e35f-13d3-4bbb-a4af-c83dba39856f-kube-api-access-gczxb\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:00.038791 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.038614    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e76e35f-13d3-4bbb-a4af-c83dba39856f-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:00.112323 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.112294    2565 generic.go:356] "Generic (PLEG): container finished" podID="7e76e35f-13d3-4bbb-a4af-c83dba39856f" containerID="c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e" exitCode=0
Jul 07 18:50:00.112486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.112339    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"
Jul 07 18:50:00.112486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.112367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt" event={"ID":"7e76e35f-13d3-4bbb-a4af-c83dba39856f","Type":"ContainerDied","Data":"c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e"}
Jul 07 18:50:00.112486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.112422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt" event={"ID":"7e76e35f-13d3-4bbb-a4af-c83dba39856f","Type":"ContainerDied","Data":"6141863a4bfefd1cf33d98a14b82c3edc674cfd3770bf3974fd8f2df2c161af9"}
Jul 07 18:50:00.112486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.112451    2565 scope.go:117] "RemoveContainer" containerID="c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e"
Jul 07 18:50:00.120756 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.120739    2565 scope.go:117] "RemoveContainer" containerID="c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e"
Jul 07 18:50:00.120995 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:00.120979    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e\": container with ID starting with c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e not found: ID does not exist" containerID="c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e"
Jul 07 18:50:00.121036 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.121002    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e"} err="failed to get container status \"c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e\": rpc error: code = NotFound desc = could not find container \"c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e\": container with ID starting with c412ea3d1ef8851e65a26e573aaf3d71d46be1521add353fb3056db2672fea6e not found: ID does not exist"
Jul 07 18:50:00.138018 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.137997    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"]
Jul 07 18:50:00.149092 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:00.149071    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-4110137c-865784f88b-2zmzt"]
Jul 07 18:50:01.047566 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:01.047539    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e76e35f-13d3-4bbb-a4af-c83dba39856f" path="/var/lib/kubelet/pods/7e76e35f-13d3-4bbb-a4af-c83dba39856f/volumes"
Jul 07 18:50:07.311516 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.311451    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"]
Jul 07 18:50:07.312000 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.311863    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e76e35f-13d3-4bbb-a4af-c83dba39856f" containerName="maas-api"
Jul 07 18:50:07.312000 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.311875    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e76e35f-13d3-4bbb-a4af-c83dba39856f" containerName="maas-api"
Jul 07 18:50:07.312000 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.311934    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e76e35f-13d3-4bbb-a4af-c83dba39856f" containerName="maas-api"
Jul 07 18:50:07.316379 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.316358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.318859 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.318838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-f6d70e44-serving-cert\""
Jul 07 18:50:07.319053 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.319040    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:50:07.336569 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.336538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"]
Jul 07 18:50:07.401359 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.401328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1239198-0382-4d5a-87b4-cf3f23454939-openshift-service-ca\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.401552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.401392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.401552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.401410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddsg\" (UniqueName: \"kubernetes.io/projected/f1239198-0382-4d5a-87b4-cf3f23454939-kube-api-access-gddsg\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.501898 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.501867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1239198-0382-4d5a-87b4-cf3f23454939-openshift-service-ca\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.502073 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.501946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.502073 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.501977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddsg\" (UniqueName: \"kubernetes.io/projected/f1239198-0382-4d5a-87b4-cf3f23454939-kube-api-access-gddsg\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.502184 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:07.502126    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f6d70e44-serving-cert: secret "maas-api-e2e-ait-f6d70e44-serving-cert" not found
Jul 07 18:50:07.502233 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:07.502222    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls podName:f1239198-0382-4d5a-87b4-cf3f23454939 nodeName:}" failed. No retries permitted until 2026-07-07 18:50:08.002199116 +0000 UTC m=+1635.577962988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls") pod "maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" (UID: "f1239198-0382-4d5a-87b4-cf3f23454939") : secret "maas-api-e2e-ait-f6d70e44-serving-cert" not found
Jul 07 18:50:07.502630 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.502613    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1239198-0382-4d5a-87b4-cf3f23454939-openshift-service-ca\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:07.518729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:07.518698    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddsg\" (UniqueName: \"kubernetes.io/projected/f1239198-0382-4d5a-87b4-cf3f23454939-kube-api-access-gddsg\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:08.006566 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:08.006532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:08.008918 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:08.008891    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls\") pod \"maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") " pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:08.226273 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:08.226245    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:08.375750 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:08.375594    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"]
Jul 07 18:50:08.378036 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:50:08.378009    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1239198_0382_4d5a_87b4_cf3f23454939.slice/crio-2158b9d221b64edfc5629338b4bc597f3c437a506d840b2d6e5332fc5e5500c4 WatchSource:0}: Error finding container 2158b9d221b64edfc5629338b4bc597f3c437a506d840b2d6e5332fc5e5500c4: Status 404 returned error can't find the container with id 2158b9d221b64edfc5629338b4bc597f3c437a506d840b2d6e5332fc5e5500c4
Jul 07 18:50:09.156279 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:09.156194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" event={"ID":"f1239198-0382-4d5a-87b4-cf3f23454939","Type":"ContainerStarted","Data":"a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21"}
Jul 07 18:50:09.156279 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:09.156258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" event={"ID":"f1239198-0382-4d5a-87b4-cf3f23454939","Type":"ContainerStarted","Data":"2158b9d221b64edfc5629338b4bc597f3c437a506d840b2d6e5332fc5e5500c4"}
Jul 07 18:50:09.156443 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:09.156377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:09.190773 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:09.190730    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" podStartSLOduration=1.7412589189999998 podStartE2EDuration="2.190716998s" totalImagesPullingTime="449.458079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:09.189518395 +0000 UTC m=+1636.765282290" watchObservedRunningTime="2026-07-07 18:50:09.190716998 +0000 UTC m=+1636.766480929"
Jul 07 18:50:12.832873 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:12.832841    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"]
Jul 07 18:50:12.833270 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:12.833043    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" podUID="f1239198-0382-4d5a-87b4-cf3f23454939" containerName="maas-api" containerID="cri-o://a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21" gracePeriod=30
Jul 07 18:50:13.074861 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.074838    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:13.149619 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.149561    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1239198-0382-4d5a-87b4-cf3f23454939-openshift-service-ca\") pod \"f1239198-0382-4d5a-87b4-cf3f23454939\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") "
Jul 07 18:50:13.149729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.149680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls\") pod \"f1239198-0382-4d5a-87b4-cf3f23454939\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") "
Jul 07 18:50:13.149729 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.149709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gddsg\" (UniqueName: \"kubernetes.io/projected/f1239198-0382-4d5a-87b4-cf3f23454939-kube-api-access-gddsg\") pod \"f1239198-0382-4d5a-87b4-cf3f23454939\" (UID: \"f1239198-0382-4d5a-87b4-cf3f23454939\") "
Jul 07 18:50:13.149944 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.149922    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1239198-0382-4d5a-87b4-cf3f23454939-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1239198-0382-4d5a-87b4-cf3f23454939" (UID: "f1239198-0382-4d5a-87b4-cf3f23454939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:50:13.151622 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.151596    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1239198-0382-4d5a-87b4-cf3f23454939-kube-api-access-gddsg" (OuterVolumeSpecName: "kube-api-access-gddsg") pod "f1239198-0382-4d5a-87b4-cf3f23454939" (UID: "f1239198-0382-4d5a-87b4-cf3f23454939"). InnerVolumeSpecName "kube-api-access-gddsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:50:13.151756 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.151595    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1239198-0382-4d5a-87b4-cf3f23454939" (UID: "f1239198-0382-4d5a-87b4-cf3f23454939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:50:13.175539 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.175512    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1239198-0382-4d5a-87b4-cf3f23454939" containerID="a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21" exitCode=0
Jul 07 18:50:13.175626 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.175558    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"
Jul 07 18:50:13.175626 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.175602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" event={"ID":"f1239198-0382-4d5a-87b4-cf3f23454939","Type":"ContainerDied","Data":"a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21"}
Jul 07 18:50:13.175707 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.175652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq" event={"ID":"f1239198-0382-4d5a-87b4-cf3f23454939","Type":"ContainerDied","Data":"2158b9d221b64edfc5629338b4bc597f3c437a506d840b2d6e5332fc5e5500c4"}
Jul 07 18:50:13.175707 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.175679    2565 scope.go:117] "RemoveContainer" containerID="a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21"
Jul 07 18:50:13.187196 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.187177    2565 scope.go:117] "RemoveContainer" containerID="a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21"
Jul 07 18:50:13.187506 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:13.187487    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21\": container with ID starting with a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21 not found: ID does not exist" containerID="a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21"
Jul 07 18:50:13.187572 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.187517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21"} err="failed to get container status \"a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21\": rpc error: code = NotFound desc = could not find container \"a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21\": container with ID starting with a56f25e2ba12241f6e71321c3414a16aef0f82ac6118dae9a5ed62afbc016c21 not found: ID does not exist"
Jul 07 18:50:13.200574 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.200551    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"]
Jul 07 18:50:13.213622 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.213601    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-f6d70e44-89d6b6d44-cscmq"]
Jul 07 18:50:13.250760 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.250738    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1239198-0382-4d5a-87b4-cf3f23454939-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:13.250760 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.250758    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddsg\" (UniqueName: \"kubernetes.io/projected/f1239198-0382-4d5a-87b4-cf3f23454939-kube-api-access-gddsg\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:13.250875 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:13.250767    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1239198-0382-4d5a-87b4-cf3f23454939-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:15.047680 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:15.047652    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1239198-0382-4d5a-87b4-cf3f23454939" path="/var/lib/kubelet/pods/f1239198-0382-4d5a-87b4-cf3f23454939/volumes"
Jul 07 18:50:26.775172 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.775138    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"]
Jul 07 18:50:26.775634 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.775521    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1239198-0382-4d5a-87b4-cf3f23454939" containerName="maas-api"
Jul 07 18:50:26.775634 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.775537    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1239198-0382-4d5a-87b4-cf3f23454939" containerName="maas-api"
Jul 07 18:50:26.775777 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.775638    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1239198-0382-4d5a-87b4-cf3f23454939" containerName="maas-api"
Jul 07 18:50:26.778714 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.778693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.782096 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.782075    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-593c94a9-serving-cert\""
Jul 07 18:50:26.782486 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.782452    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:50:26.802608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.802587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"]
Jul 07 18:50:26.867885 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.867854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7551b9-b661-4162-b14c-9c00e6fe300d-openshift-service-ca\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.868027 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.867893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.868027 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.867974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkl6\" (UniqueName: \"kubernetes.io/projected/0b7551b9-b661-4162-b14c-9c00e6fe300d-kube-api-access-xpkl6\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.969228 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.969197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkl6\" (UniqueName: \"kubernetes.io/projected/0b7551b9-b661-4162-b14c-9c00e6fe300d-kube-api-access-xpkl6\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.969358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.969249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7551b9-b661-4162-b14c-9c00e6fe300d-openshift-service-ca\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.969358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.969343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.975838 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.970394    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7551b9-b661-4162-b14c-9c00e6fe300d-openshift-service-ca\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:26.975952 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:26.975921    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-593c94a9-serving-cert: secret "maas-api-e2e-derive-593c94a9-serving-cert" not found
Jul 07 18:50:26.976125 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:26.976096    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls podName:0b7551b9-b661-4162-b14c-9c00e6fe300d nodeName:}" failed. No retries permitted until 2026-07-07 18:50:27.476022254 +0000 UTC m=+1655.051786127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls") pod "maas-api-e2e-derive-593c94a9-54787754cb-x655g" (UID: "0b7551b9-b661-4162-b14c-9c00e6fe300d") : secret "maas-api-e2e-derive-593c94a9-serving-cert" not found
Jul 07 18:50:26.987055 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:26.987033    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkl6\" (UniqueName: \"kubernetes.io/projected/0b7551b9-b661-4162-b14c-9c00e6fe300d-kube-api-access-xpkl6\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:27.573560 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:27.573531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:27.575882 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:27.575850    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls\") pod \"maas-api-e2e-derive-593c94a9-54787754cb-x655g\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") " pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:27.688725 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:27.688695    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:27.824838 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:27.824787    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"]
Jul 07 18:50:27.825914 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:50:27.825892    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7551b9_b661_4162_b14c_9c00e6fe300d.slice/crio-0310601219d2ad0f793d35a2c37b81d810ee017bb023858c13e4188ea6c3ae77 WatchSource:0}: Error finding container 0310601219d2ad0f793d35a2c37b81d810ee017bb023858c13e4188ea6c3ae77: Status 404 returned error can't find the container with id 0310601219d2ad0f793d35a2c37b81d810ee017bb023858c13e4188ea6c3ae77
Jul 07 18:50:28.244345 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:28.244265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g" event={"ID":"0b7551b9-b661-4162-b14c-9c00e6fe300d","Type":"ContainerStarted","Data":"0310601219d2ad0f793d35a2c37b81d810ee017bb023858c13e4188ea6c3ae77"}
Jul 07 18:50:29.250924 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:29.250834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g" event={"ID":"0b7551b9-b661-4162-b14c-9c00e6fe300d","Type":"ContainerStarted","Data":"0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38"}
Jul 07 18:50:29.251280 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:29.250963    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:29.274607 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:29.274567    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g" podStartSLOduration=2.7880989080000003 podStartE2EDuration="3.274553263s" totalImagesPullingTime="486.454355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:29.272386712 +0000 UTC m=+1656.848150617" watchObservedRunningTime="2026-07-07 18:50:29.274553263 +0000 UTC m=+1656.850317155"
Jul 07 18:50:31.723006 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:31.722967    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"]
Jul 07 18:50:31.723566 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:31.723179    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g" podUID="0b7551b9-b661-4162-b14c-9c00e6fe300d" containerName="maas-api" containerID="cri-o://0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38" gracePeriod=30
Jul 07 18:50:31.957982 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:31.957961    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:32.117160 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.117134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls\") pod \"0b7551b9-b661-4162-b14c-9c00e6fe300d\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") "
Jul 07 18:50:32.117310 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.117183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7551b9-b661-4162-b14c-9c00e6fe300d-openshift-service-ca\") pod \"0b7551b9-b661-4162-b14c-9c00e6fe300d\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") "
Jul 07 18:50:32.117310 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.117207    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpkl6\" (UniqueName: \"kubernetes.io/projected/0b7551b9-b661-4162-b14c-9c00e6fe300d-kube-api-access-xpkl6\") pod \"0b7551b9-b661-4162-b14c-9c00e6fe300d\" (UID: \"0b7551b9-b661-4162-b14c-9c00e6fe300d\") "
Jul 07 18:50:32.117608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.117574    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7551b9-b661-4162-b14c-9c00e6fe300d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b7551b9-b661-4162-b14c-9c00e6fe300d" (UID: "0b7551b9-b661-4162-b14c-9c00e6fe300d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:50:32.119124 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.119098    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7551b9-b661-4162-b14c-9c00e6fe300d-kube-api-access-xpkl6" (OuterVolumeSpecName: "kube-api-access-xpkl6") pod "0b7551b9-b661-4162-b14c-9c00e6fe300d" (UID: "0b7551b9-b661-4162-b14c-9c00e6fe300d"). InnerVolumeSpecName "kube-api-access-xpkl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:50:32.119238 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.119221    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b7551b9-b661-4162-b14c-9c00e6fe300d" (UID: "0b7551b9-b661-4162-b14c-9c00e6fe300d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:50:32.218076 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.218056    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7551b9-b661-4162-b14c-9c00e6fe300d-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:32.218180 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.218079    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7551b9-b661-4162-b14c-9c00e6fe300d-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:32.218180 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.218093    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpkl6\" (UniqueName: \"kubernetes.io/projected/0b7551b9-b661-4162-b14c-9c00e6fe300d-kube-api-access-xpkl6\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:50:32.265328 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.265298    2565 generic.go:356] "Generic (PLEG): container finished" podID="0b7551b9-b661-4162-b14c-9c00e6fe300d" containerID="0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38" exitCode=0
Jul 07 18:50:32.265439 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.265340    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"
Jul 07 18:50:32.265439 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.265357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g" event={"ID":"0b7551b9-b661-4162-b14c-9c00e6fe300d","Type":"ContainerDied","Data":"0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38"}
Jul 07 18:50:32.265439 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.265413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g" event={"ID":"0b7551b9-b661-4162-b14c-9c00e6fe300d","Type":"ContainerDied","Data":"0310601219d2ad0f793d35a2c37b81d810ee017bb023858c13e4188ea6c3ae77"}
Jul 07 18:50:32.265628 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.265442    2565 scope.go:117] "RemoveContainer" containerID="0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38"
Jul 07 18:50:32.274072 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.274056    2565 scope.go:117] "RemoveContainer" containerID="0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38"
Jul 07 18:50:32.274335 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:50:32.274317    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38\": container with ID starting with 0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38 not found: ID does not exist" containerID="0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38"
Jul 07 18:50:32.274390 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.274340    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38"} err="failed to get container status \"0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38\": rpc error: code = NotFound desc = could not find container \"0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38\": container with ID starting with 0dafcb1779c16a8884412154f3ac1f649d51538de3c9fe584f625d61f0725c38 not found: ID does not exist"
Jul 07 18:50:32.293720 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.293696    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"]
Jul 07 18:50:32.301020 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:32.301002    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-593c94a9-54787754cb-x655g"]
Jul 07 18:50:33.047315 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:50:33.047238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7551b9-b661-4162-b14c-9c00e6fe300d" path="/var/lib/kubelet/pods/0b7551b9-b661-4162-b14c-9c00e6fe300d/volumes"
Jul 07 18:52:21.974728 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.974692    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"]
Jul 07 18:52:21.975203 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.975182    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7551b9-b661-4162-b14c-9c00e6fe300d" containerName="maas-api"
Jul 07 18:52:21.975203 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.975203    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7551b9-b661-4162-b14c-9c00e6fe300d" containerName="maas-api"
Jul 07 18:52:21.975355 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.975313    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7551b9-b661-4162-b14c-9c00e6fe300d" containerName="maas-api"
Jul 07 18:52:21.978738 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.978719    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:21.981862 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.981840    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-8c3df14f-serving-cert\""
Jul 07 18:52:21.981975 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:21.981905    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:52:22.019738 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.019698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"]
Jul 07 18:52:22.067982 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.067942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmknd\" (UniqueName: \"kubernetes.io/projected/13990358-9df3-4483-bb31-7836474e7b8a-kube-api-access-tmknd\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.068088 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.068066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13990358-9df3-4483-bb31-7836474e7b8a-maas-api-tls\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.068144 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.068097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13990358-9df3-4483-bb31-7836474e7b8a-openshift-service-ca\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.168811 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.168781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmknd\" (UniqueName: \"kubernetes.io/projected/13990358-9df3-4483-bb31-7836474e7b8a-kube-api-access-tmknd\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.168959 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.168898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13990358-9df3-4483-bb31-7836474e7b8a-maas-api-tls\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.168959 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.168949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13990358-9df3-4483-bb31-7836474e7b8a-openshift-service-ca\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.169668 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.169645    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13990358-9df3-4483-bb31-7836474e7b8a-openshift-service-ca\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.171210 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.171190    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13990358-9df3-4483-bb31-7836474e7b8a-maas-api-tls\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.178619 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.178591    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmknd\" (UniqueName: \"kubernetes.io/projected/13990358-9df3-4483-bb31-7836474e7b8a-kube-api-access-tmknd\") pod \"maas-api-e2e-mt-8c3df14f-7b45654956-drxjr\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") " pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.289149 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.289089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:22.423123 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.423092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"]
Jul 07 18:52:22.424309 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:52:22.424285    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13990358_9df3_4483_bb31_7836474e7b8a.slice/crio-b310d7bd451feffb5388efb9f4777f1ae0df665680d329966546e4e822fbfd86 WatchSource:0}: Error finding container b310d7bd451feffb5388efb9f4777f1ae0df665680d329966546e4e822fbfd86: Status 404 returned error can't find the container with id b310d7bd451feffb5388efb9f4777f1ae0df665680d329966546e4e822fbfd86
Jul 07 18:52:22.764364 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:22.764329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr" event={"ID":"13990358-9df3-4483-bb31-7836474e7b8a","Type":"ContainerStarted","Data":"b310d7bd451feffb5388efb9f4777f1ae0df665680d329966546e4e822fbfd86"}
Jul 07 18:52:23.770316 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:23.770283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr" event={"ID":"13990358-9df3-4483-bb31-7836474e7b8a","Type":"ContainerStarted","Data":"16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df"}
Jul 07 18:52:23.770710 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:23.770366    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:23.795081 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:23.795036    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr" podStartSLOduration=2.339226888 podStartE2EDuration="2.79502161s" totalImagesPullingTime="455.794722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:52:23.792906172 +0000 UTC m=+1771.368670066" watchObservedRunningTime="2026-07-07 18:52:23.79502161 +0000 UTC m=+1771.370785503"
Jul 07 18:52:29.779624 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:29.779595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:35.947360 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:35.947326    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"]
Jul 07 18:52:35.947786 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:35.947542    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr" podUID="13990358-9df3-4483-bb31-7836474e7b8a" containerName="maas-api" containerID="cri-o://16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df" gracePeriod=30
Jul 07 18:52:36.191370 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.191347    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:36.286663 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.286601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13990358-9df3-4483-bb31-7836474e7b8a-openshift-service-ca\") pod \"13990358-9df3-4483-bb31-7836474e7b8a\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") "
Jul 07 18:52:36.286663 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.286642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13990358-9df3-4483-bb31-7836474e7b8a-maas-api-tls\") pod \"13990358-9df3-4483-bb31-7836474e7b8a\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") "
Jul 07 18:52:36.286844 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.286675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmknd\" (UniqueName: \"kubernetes.io/projected/13990358-9df3-4483-bb31-7836474e7b8a-kube-api-access-tmknd\") pod \"13990358-9df3-4483-bb31-7836474e7b8a\" (UID: \"13990358-9df3-4483-bb31-7836474e7b8a\") "
Jul 07 18:52:36.286988 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.286964    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13990358-9df3-4483-bb31-7836474e7b8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13990358-9df3-4483-bb31-7836474e7b8a" (UID: "13990358-9df3-4483-bb31-7836474e7b8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:52:36.288753 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.288723    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13990358-9df3-4483-bb31-7836474e7b8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13990358-9df3-4483-bb31-7836474e7b8a" (UID: "13990358-9df3-4483-bb31-7836474e7b8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:52:36.288851 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.288761    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13990358-9df3-4483-bb31-7836474e7b8a-kube-api-access-tmknd" (OuterVolumeSpecName: "kube-api-access-tmknd") pod "13990358-9df3-4483-bb31-7836474e7b8a" (UID: "13990358-9df3-4483-bb31-7836474e7b8a"). InnerVolumeSpecName "kube-api-access-tmknd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:52:36.387246 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.387223    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13990358-9df3-4483-bb31-7836474e7b8a-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:52:36.387246 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.387246    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13990358-9df3-4483-bb31-7836474e7b8a-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:52:36.387395 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.387257    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmknd\" (UniqueName: \"kubernetes.io/projected/13990358-9df3-4483-bb31-7836474e7b8a-kube-api-access-tmknd\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:52:36.828506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.828452    2565 generic.go:356] "Generic (PLEG): container finished" podID="13990358-9df3-4483-bb31-7836474e7b8a" containerID="16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df" exitCode=0
Jul 07 18:52:36.828659 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.828518    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"
Jul 07 18:52:36.828659 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.828539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr" event={"ID":"13990358-9df3-4483-bb31-7836474e7b8a","Type":"ContainerDied","Data":"16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df"}
Jul 07 18:52:36.828659 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.828581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr" event={"ID":"13990358-9df3-4483-bb31-7836474e7b8a","Type":"ContainerDied","Data":"b310d7bd451feffb5388efb9f4777f1ae0df665680d329966546e4e822fbfd86"}
Jul 07 18:52:36.828659 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.828603    2565 scope.go:117] "RemoveContainer" containerID="16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df"
Jul 07 18:52:36.837682 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.837665    2565 scope.go:117] "RemoveContainer" containerID="16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df"
Jul 07 18:52:36.837922 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:52:36.837903    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df\": container with ID starting with 16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df not found: ID does not exist" containerID="16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df"
Jul 07 18:52:36.837961 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.837931    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df"} err="failed to get container status \"16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df\": rpc error: code = NotFound desc = could not find container \"16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df\": container with ID starting with 16aab8c1fee781272a0b79bf8fdc6c7cab0b2cd1ff51952683aba415ca0fb2df not found: ID does not exist"
Jul 07 18:52:36.853636 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.853615    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"]
Jul 07 18:52:36.861504 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:36.861486    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-8c3df14f-7b45654956-drxjr"]
Jul 07 18:52:37.047309 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:52:37.047287    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13990358-9df3-4483-bb31-7836474e7b8a" path="/var/lib/kubelet/pods/13990358-9df3-4483-bb31-7836474e7b8a/volumes"
Jul 07 18:53:11.163092 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.163060    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"]
Jul 07 18:53:11.163670 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.163651    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13990358-9df3-4483-bb31-7836474e7b8a" containerName="maas-api"
Jul 07 18:53:11.163792 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.163673    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13990358-9df3-4483-bb31-7836474e7b8a" containerName="maas-api"
Jul 07 18:53:11.163792 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.163780    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13990358-9df3-4483-bb31-7836474e7b8a" containerName="maas-api"
Jul 07 18:53:11.166884 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.166861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.169785 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.169760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:53:11.169908 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.169762    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-3ae90c5b-serving-cert\""
Jul 07 18:53:11.180736 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.180713    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"]
Jul 07 18:53:11.261439 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.261411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-openshift-service-ca\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.261619 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.261507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmb5m\" (UniqueName: \"kubernetes.io/projected/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-kube-api-access-tmb5m\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.261619 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.261570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.362797 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.362773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmb5m\" (UniqueName: \"kubernetes.io/projected/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-kube-api-access-tmb5m\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.362910 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.362830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.362910 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.362875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-openshift-service-ca\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.362982 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:53:11.362912    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-3ae90c5b-serving-cert: secret "maas-api-e2e-mt-3ae90c5b-serving-cert" not found
Jul 07 18:53:11.362982 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:53:11.362967    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls podName:3d80e5bf-e26e-492e-bfb7-04d4916b4b13 nodeName:}" failed. No retries permitted until 2026-07-07 18:53:11.862952593 +0000 UTC m=+1819.438716465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls") pod "maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" (UID: "3d80e5bf-e26e-492e-bfb7-04d4916b4b13") : secret "maas-api-e2e-mt-3ae90c5b-serving-cert" not found
Jul 07 18:53:11.363434 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.363418    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-openshift-service-ca\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.380767 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.380737    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmb5m\" (UniqueName: \"kubernetes.io/projected/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-kube-api-access-tmb5m\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.866764 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.866736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:11.869188 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:11.869161    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls\") pod \"maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") " pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:12.081288 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:12.081263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:12.218482 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:12.218435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"]
Jul 07 18:53:12.220201 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:53:12.220176    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d80e5bf_e26e_492e_bfb7_04d4916b4b13.slice/crio-a36ac62df073d7fb724c550323f8f87f6fe9db2f85d29bd315b881566e32db99 WatchSource:0}: Error finding container a36ac62df073d7fb724c550323f8f87f6fe9db2f85d29bd315b881566e32db99: Status 404 returned error can't find the container with id a36ac62df073d7fb724c550323f8f87f6fe9db2f85d29bd315b881566e32db99
Jul 07 18:53:12.993721 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:12.993691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" event={"ID":"3d80e5bf-e26e-492e-bfb7-04d4916b4b13","Type":"ContainerStarted","Data":"24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b"}
Jul 07 18:53:12.993851 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:12.993740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" event={"ID":"3d80e5bf-e26e-492e-bfb7-04d4916b4b13","Type":"ContainerStarted","Data":"a36ac62df073d7fb724c550323f8f87f6fe9db2f85d29bd315b881566e32db99"}
Jul 07 18:53:12.993851 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:12.993822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:19.003171 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:19.003140    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:19.042016 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:19.041959    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" podStartSLOduration=7.49236457 podStartE2EDuration="8.041945198s" totalImagesPullingTime="549.580628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:53:13.031200527 +0000 UTC m=+1820.606964455" watchObservedRunningTime="2026-07-07 18:53:19.041945198 +0000 UTC m=+1826.617709090"
Jul 07 18:53:49.591909 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.591876    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"]
Jul 07 18:53:49.592452 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.592163    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" podUID="3d80e5bf-e26e-492e-bfb7-04d4916b4b13" containerName="maas-api" containerID="cri-o://24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b" gracePeriod=30
Jul 07 18:53:49.834751 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.834724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:49.964895 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.964815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-openshift-service-ca\") pod \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") "
Jul 07 18:53:49.964895 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.964866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmb5m\" (UniqueName: \"kubernetes.io/projected/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-kube-api-access-tmb5m\") pod \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") "
Jul 07 18:53:49.964895 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.964896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls\") pod \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\" (UID: \"3d80e5bf-e26e-492e-bfb7-04d4916b4b13\") "
Jul 07 18:53:49.965150 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.965128    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d80e5bf-e26e-492e-bfb7-04d4916b4b13" (UID: "3d80e5bf-e26e-492e-bfb7-04d4916b4b13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:53:49.966935 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.966905    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d80e5bf-e26e-492e-bfb7-04d4916b4b13" (UID: "3d80e5bf-e26e-492e-bfb7-04d4916b4b13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:53:49.966935 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:49.966906    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-kube-api-access-tmb5m" (OuterVolumeSpecName: "kube-api-access-tmb5m") pod "3d80e5bf-e26e-492e-bfb7-04d4916b4b13" (UID: "3d80e5bf-e26e-492e-bfb7-04d4916b4b13"). InnerVolumeSpecName "kube-api-access-tmb5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:53:50.065342 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.065316    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:53:50.065342 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.065336    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:53:50.065497 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.065349    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmb5m\" (UniqueName: \"kubernetes.io/projected/3d80e5bf-e26e-492e-bfb7-04d4916b4b13-kube-api-access-tmb5m\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:53:50.163528 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.163501    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d80e5bf-e26e-492e-bfb7-04d4916b4b13" containerID="24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b" exitCode=0
Jul 07 18:53:50.163646 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.163541    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"
Jul 07 18:53:50.163646 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.163589    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" event={"ID":"3d80e5bf-e26e-492e-bfb7-04d4916b4b13","Type":"ContainerDied","Data":"24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b"}
Jul 07 18:53:50.163646 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.163634    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv" event={"ID":"3d80e5bf-e26e-492e-bfb7-04d4916b4b13","Type":"ContainerDied","Data":"a36ac62df073d7fb724c550323f8f87f6fe9db2f85d29bd315b881566e32db99"}
Jul 07 18:53:50.163837 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.163656    2565 scope.go:117] "RemoveContainer" containerID="24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b"
Jul 07 18:53:50.172270 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.172253    2565 scope.go:117] "RemoveContainer" containerID="24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b"
Jul 07 18:53:50.172535 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:53:50.172511    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b\": container with ID starting with 24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b not found: ID does not exist" containerID="24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b"
Jul 07 18:53:50.172636 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.172539    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b"} err="failed to get container status \"24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b\": rpc error: code = NotFound desc = could not find container \"24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b\": container with ID starting with 24e3ce103720d12b0028aba34dd57295d88320708aaf70726fcb907d9287145b not found: ID does not exist"
Jul 07 18:53:50.190014 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.189994    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"]
Jul 07 18:53:50.192849 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:50.192823    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ae90c5b-864ddd5759-jcxpv"]
Jul 07 18:53:51.047404 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:53:51.047375    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d80e5bf-e26e-492e-bfb7-04d4916b4b13" path="/var/lib/kubelet/pods/3d80e5bf-e26e-492e-bfb7-04d4916b4b13/volumes"
Jul 07 18:56:11.529016 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.528930    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"]
Jul 07 18:56:11.529475 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.529320    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d80e5bf-e26e-492e-bfb7-04d4916b4b13" containerName="maas-api"
Jul 07 18:56:11.529475 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.529333    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d80e5bf-e26e-492e-bfb7-04d4916b4b13" containerName="maas-api"
Jul 07 18:56:11.529475 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.529420    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d80e5bf-e26e-492e-bfb7-04d4916b4b13" containerName="maas-api"
Jul 07 18:56:11.532526 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.532506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.536757 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.536717    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-b07e06-serving-cert\""
Jul 07 18:56:11.536757 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.536747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:56:11.546750 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.546730    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"]
Jul 07 18:56:11.655194 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.655173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.655302 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.655213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.655302 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.655242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsdc\" (UniqueName: \"kubernetes.io/projected/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-kube-api-access-bhsdc\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.755632 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.755608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.755739 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.755644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsdc\" (UniqueName: \"kubernetes.io/projected/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-kube-api-access-bhsdc\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.755739 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.755692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.755861 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:56:11.755805    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-b07e06-serving-cert: secret "maas-api-e2e-inf-a-b07e06-serving-cert" not found
Jul 07 18:56:11.755861 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:56:11.755861    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls podName:b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e nodeName:}" failed. No retries permitted until 2026-07-07 18:56:12.255846016 +0000 UTC m=+1999.831609889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls") pod "maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" (UID: "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e") : secret "maas-api-e2e-inf-a-b07e06-serving-cert" not found
Jul 07 18:56:11.756270 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.756252    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:11.775018 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:11.774990    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsdc\" (UniqueName: \"kubernetes.io/projected/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-kube-api-access-bhsdc\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:12.260621 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:12.260589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:12.262888 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:12.262864    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") " pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:12.443908 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:12.443881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:12.591112 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:12.591092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"]
Jul 07 18:56:12.596255 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:56:12.596221    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81f3c24_e6ea_4c20_91eb_06b52b5b6a8e.slice/crio-d06de72c67c11320f3591d09d81737ea7f2271a438b4ac17959b3f53a8e58695 WatchSource:0}: Error finding container d06de72c67c11320f3591d09d81737ea7f2271a438b4ac17959b3f53a8e58695: Status 404 returned error can't find the container with id d06de72c67c11320f3591d09d81737ea7f2271a438b4ac17959b3f53a8e58695
Jul 07 18:56:12.598183 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:12.598163    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:56:12.823095 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:12.823067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" event={"ID":"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e","Type":"ContainerStarted","Data":"d06de72c67c11320f3591d09d81737ea7f2271a438b4ac17959b3f53a8e58695"}
Jul 07 18:56:13.828875 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:13.828835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" event={"ID":"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e","Type":"ContainerStarted","Data":"45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d"}
Jul 07 18:56:13.829247 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:13.828966    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:13.856235 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:13.856189    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" podStartSLOduration=2.356409716 podStartE2EDuration="2.8561745s" totalImagesPullingTime="499.764784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:56:13.854573208 +0000 UTC m=+2001.430337101" watchObservedRunningTime="2026-07-07 18:56:13.8561745 +0000 UTC m=+2001.431938410"
Jul 07 18:56:19.839105 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:19.839074    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:56:20.231964 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.231887    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"]
Jul 07 18:56:20.235703 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.235676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.238165 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.238141    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-c5164d-serving-cert\""
Jul 07 18:56:20.250358 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.250331    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"]
Jul 07 18:56:20.430405 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.430376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpml\" (UniqueName: \"kubernetes.io/projected/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-kube-api-access-vvpml\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.430578 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.430413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-maas-api-tls\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.430578 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.430495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.531593 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.531528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.531593 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.531573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpml\" (UniqueName: \"kubernetes.io/projected/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-kube-api-access-vvpml\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.531809 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.531596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-maas-api-tls\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.532173 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.532155    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.533903 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.533883    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-maas-api-tls\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.548029 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.548005    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpml\" (UniqueName: \"kubernetes.io/projected/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-kube-api-access-vvpml\") pod \"maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") " pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:20.846869 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:20.846840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:21.013075 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:21.013047    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"]
Jul 07 18:56:21.015001 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:56:21.014970    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c5abcd_b0b1_4b80_a199_3d8e9d2d58a8.slice/crio-6a5b11e38e1f635cab8ed1a9b5c9492a456e4df5472d07fc6ce320a52a3c5dce WatchSource:0}: Error finding container 6a5b11e38e1f635cab8ed1a9b5c9492a456e4df5472d07fc6ce320a52a3c5dce: Status 404 returned error can't find the container with id 6a5b11e38e1f635cab8ed1a9b5c9492a456e4df5472d07fc6ce320a52a3c5dce
Jul 07 18:56:21.866853 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:21.866826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq" event={"ID":"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8","Type":"ContainerStarted","Data":"f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3"}
Jul 07 18:56:21.867233 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:21.866875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq" event={"ID":"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8","Type":"ContainerStarted","Data":"6a5b11e38e1f635cab8ed1a9b5c9492a456e4df5472d07fc6ce320a52a3c5dce"}
Jul 07 18:56:21.867233 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:21.866958    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:56:21.890166 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:21.890119    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq" podStartSLOduration=1.536994377 podStartE2EDuration="1.890106112s" totalImagesPullingTime="353.111735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:56:21.887755186 +0000 UTC m=+2009.463519080" watchObservedRunningTime="2026-07-07 18:56:21.890106112 +0000 UTC m=+2009.465870005"
Jul 07 18:56:27.875877 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:56:27.875848    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:57:19.198203 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.198118    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"]
Jul 07 18:57:19.201902 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.201880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.208360 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.208004    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c5164d\"/\"default-dockercfg-5fqbv\""
Jul 07 18:57:19.209658 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.209637    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c5164d\"/\"kube-root-ca.crt\""
Jul 07 18:57:19.209757 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.209637    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c5164d\"/\"openshift-service-ca.crt\""
Jul 07 18:57:19.210809 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.210793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c5164d\"/\"test-model-c5164d-kserve-self-signed-certs\""
Jul 07 18:57:19.239498 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.239477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"]
Jul 07 18:57:19.291250 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.291223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-model-cache\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.291333 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.291255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-home\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.291333 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.291296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-tmp-dir\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.291333 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.291323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cf7d72-4356-42a3-8e11-d618ce117798-tls-certs\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.291437 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.291372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thr27\" (UniqueName: \"kubernetes.io/projected/13cf7d72-4356-42a3-8e11-d618ce117798-kube-api-access-thr27\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.291437 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.291395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-dshm\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392254 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thr27\" (UniqueName: \"kubernetes.io/projected/13cf7d72-4356-42a3-8e11-d618ce117798-kube-api-access-thr27\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392351 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-dshm\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392351 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-model-cache\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-home\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392423 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-tmp-dir\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392538 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cf7d72-4356-42a3-8e11-d618ce117798-tls-certs\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392771 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392747    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-home\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392887 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392825    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-tmp-dir\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.392887 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.392842    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-model-cache\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.394883 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.394861    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-dshm\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.395042 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.395021    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cf7d72-4356-42a3-8e11-d618ce117798-tls-certs\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.404437 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.404416    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thr27\" (UniqueName: \"kubernetes.io/projected/13cf7d72-4356-42a3-8e11-d618ce117798-kube-api-access-thr27\") pod \"test-model-c5164d-kserve-7949956895-9lnsm\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") " pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.510808 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.510742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:19.640843 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:19.640709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"]
Jul 07 18:57:19.643547 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:57:19.643522    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13cf7d72_4356_42a3_8e11_d618ce117798.slice/crio-f8fb21bc34581353fd16e747ae4aab09961ecb964ebb0ef2ff3d5f7b9f281e9e WatchSource:0}: Error finding container f8fb21bc34581353fd16e747ae4aab09961ecb964ebb0ef2ff3d5f7b9f281e9e: Status 404 returned error can't find the container with id f8fb21bc34581353fd16e747ae4aab09961ecb964ebb0ef2ff3d5f7b9f281e9e
Jul 07 18:57:20.138979 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:20.138943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm" event={"ID":"13cf7d72-4356-42a3-8e11-d618ce117798","Type":"ContainerStarted","Data":"0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013"}
Jul 07 18:57:20.138979 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:20.138985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm" event={"ID":"13cf7d72-4356-42a3-8e11-d618ce117798","Type":"ContainerStarted","Data":"f8fb21bc34581353fd16e747ae4aab09961ecb964ebb0ef2ff3d5f7b9f281e9e"}
Jul 07 18:57:20.162631 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:20.162579    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm" podStartSLOduration=1.162564519 podStartE2EDuration="1.162564519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:57:20.162480385 +0000 UTC m=+2067.738244291" watchObservedRunningTime="2026-07-07 18:57:20.162564519 +0000 UTC m=+2067.738328411"
Jul 07 18:57:29.511300 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:29.511259    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:29.511959 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:29.511314    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:29.523886 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:29.523861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:30.193096 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:30.193067    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:40.801190 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:40.801158    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"]
Jul 07 18:57:42.142079 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.142049    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"]
Jul 07 18:57:42.142469 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.142271    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" podUID="b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" containerName="maas-api" containerID="cri-o://45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d" gracePeriod=30
Jul 07 18:57:42.389507 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.389484    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:57:42.473226 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.473173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls\") pod \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") "
Jul 07 18:57:42.473346 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.473228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhsdc\" (UniqueName: \"kubernetes.io/projected/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-kube-api-access-bhsdc\") pod \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") "
Jul 07 18:57:42.473346 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.473289    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-openshift-service-ca\") pod \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\" (UID: \"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e\") "
Jul 07 18:57:42.473674 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.473646    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" (UID: "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:57:42.475075 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.475046    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" (UID: "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:57:42.475230 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.475212    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-kube-api-access-bhsdc" (OuterVolumeSpecName: "kube-api-access-bhsdc") pod "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" (UID: "b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e"). InnerVolumeSpecName "kube-api-access-bhsdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:57:42.574399 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.574373    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:42.574399 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.574395    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhsdc\" (UniqueName: \"kubernetes.io/projected/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-kube-api-access-bhsdc\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:42.574555 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:42.574405    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:43.240089 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.240055    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" containerID="45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d" exitCode=0
Jul 07 18:57:43.240479 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.240103    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"
Jul 07 18:57:43.240479 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.240141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" event={"ID":"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e","Type":"ContainerDied","Data":"45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d"}
Jul 07 18:57:43.240479 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.240195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf" event={"ID":"b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e","Type":"ContainerDied","Data":"d06de72c67c11320f3591d09d81737ea7f2271a438b4ac17959b3f53a8e58695"}
Jul 07 18:57:43.240479 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.240219    2565 scope.go:117] "RemoveContainer" containerID="45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d"
Jul 07 18:57:43.248734 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.248707    2565 scope.go:117] "RemoveContainer" containerID="45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d"
Jul 07 18:57:43.249006 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:57:43.248988    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d\": container with ID starting with 45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d not found: ID does not exist" containerID="45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d"
Jul 07 18:57:43.249055 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.249014    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d"} err="failed to get container status \"45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d\": rpc error: code = NotFound desc = could not find container \"45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d\": container with ID starting with 45dfd94fb817866d0deb30f7ba9d81ad0fe9c137377fd84a4cb6faa56b441f6d not found: ID does not exist"
Jul 07 18:57:43.262450 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.262426    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"]
Jul 07 18:57:43.265931 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:43.265912    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-b07e06-695f85f5cc-jvzhf"]
Jul 07 18:57:45.048165 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:45.048122    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" path="/var/lib/kubelet/pods/b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e/volumes"
Jul 07 18:57:55.820754 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:55.820692    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm" podUID="13cf7d72-4356-42a3-8e11-d618ce117798" containerName="main" containerID="cri-o://0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013" gracePeriod=45
Jul 07 18:57:56.070351 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.070328    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:56.195849 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.195783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-model-cache\") pod \"13cf7d72-4356-42a3-8e11-d618ce117798\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") "
Jul 07 18:57:56.195849 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.195835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-tmp-dir\") pod \"13cf7d72-4356-42a3-8e11-d618ce117798\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") "
Jul 07 18:57:56.195999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.195854    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thr27\" (UniqueName: \"kubernetes.io/projected/13cf7d72-4356-42a3-8e11-d618ce117798-kube-api-access-thr27\") pod \"13cf7d72-4356-42a3-8e11-d618ce117798\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") "
Jul 07 18:57:56.195999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.195872    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-home\") pod \"13cf7d72-4356-42a3-8e11-d618ce117798\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") "
Jul 07 18:57:56.195999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.195914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-dshm\") pod \"13cf7d72-4356-42a3-8e11-d618ce117798\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") "
Jul 07 18:57:56.195999 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.195944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cf7d72-4356-42a3-8e11-d618ce117798-tls-certs\") pod \"13cf7d72-4356-42a3-8e11-d618ce117798\" (UID: \"13cf7d72-4356-42a3-8e11-d618ce117798\") "
Jul 07 18:57:56.196184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.196070    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-model-cache" (OuterVolumeSpecName: "model-cache") pod "13cf7d72-4356-42a3-8e11-d618ce117798" (UID: "13cf7d72-4356-42a3-8e11-d618ce117798"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:57:56.196184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.196087    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13cf7d72-4356-42a3-8e11-d618ce117798" (UID: "13cf7d72-4356-42a3-8e11-d618ce117798"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:57:56.196184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.196175    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-home" (OuterVolumeSpecName: "home") pod "13cf7d72-4356-42a3-8e11-d618ce117798" (UID: "13cf7d72-4356-42a3-8e11-d618ce117798"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:57:56.196360 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.196274    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-tmp-dir\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:56.196360 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.196287    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-home\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:56.196360 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.196295    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-model-cache\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:56.198029 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.197981    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-dshm" (OuterVolumeSpecName: "dshm") pod "13cf7d72-4356-42a3-8e11-d618ce117798" (UID: "13cf7d72-4356-42a3-8e11-d618ce117798"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:57:56.198133 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.198018    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13cf7d72-4356-42a3-8e11-d618ce117798-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13cf7d72-4356-42a3-8e11-d618ce117798" (UID: "13cf7d72-4356-42a3-8e11-d618ce117798"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:57:56.198133 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.198056    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cf7d72-4356-42a3-8e11-d618ce117798-kube-api-access-thr27" (OuterVolumeSpecName: "kube-api-access-thr27") pod "13cf7d72-4356-42a3-8e11-d618ce117798" (UID: "13cf7d72-4356-42a3-8e11-d618ce117798"). InnerVolumeSpecName "kube-api-access-thr27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:57:56.296802 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.296778    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thr27\" (UniqueName: \"kubernetes.io/projected/13cf7d72-4356-42a3-8e11-d618ce117798-kube-api-access-thr27\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:56.296802 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.296798    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cf7d72-4356-42a3-8e11-d618ce117798-dshm\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:56.296967 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.296808    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cf7d72-4356-42a3-8e11-d618ce117798-tls-certs\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:57:56.300684 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.300662    2565 generic.go:356] "Generic (PLEG): container finished" podID="13cf7d72-4356-42a3-8e11-d618ce117798" containerID="0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013" exitCode=0
Jul 07 18:57:56.300774 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.300734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm" event={"ID":"13cf7d72-4356-42a3-8e11-d618ce117798","Type":"ContainerDied","Data":"0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013"}
Jul 07 18:57:56.300774 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.300760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm" event={"ID":"13cf7d72-4356-42a3-8e11-d618ce117798","Type":"ContainerDied","Data":"f8fb21bc34581353fd16e747ae4aab09961ecb964ebb0ef2ff3d5f7b9f281e9e"}
Jul 07 18:57:56.300774 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.300764    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"
Jul 07 18:57:56.300891 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.300775    2565 scope.go:117] "RemoveContainer" containerID="0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013"
Jul 07 18:57:56.308673 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.308656    2565 scope.go:117] "RemoveContainer" containerID="0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013"
Jul 07 18:57:56.308913 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:57:56.308895    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013\": container with ID starting with 0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013 not found: ID does not exist" containerID="0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013"
Jul 07 18:57:56.308968 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.308925    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013"} err="failed to get container status \"0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013\": rpc error: code = NotFound desc = could not find container \"0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013\": container with ID starting with 0443fb0192bb9fbc35ad45985c35bdfae72e4672a82bf210571d91ffce608013 not found: ID does not exist"
Jul 07 18:57:56.329575 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.329553    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"]
Jul 07 18:57:56.331495 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:56.331476    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c5164d/test-model-c5164d-kserve-7949956895-9lnsm"]
Jul 07 18:57:57.047518 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:57:57.047490    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13cf7d72-4356-42a3-8e11-d618ce117798" path="/var/lib/kubelet/pods/13cf7d72-4356-42a3-8e11-d618ce117798/volumes"
Jul 07 18:58:26.058123 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.058092    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"]
Jul 07 18:58:26.058648 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.058263    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq" podUID="24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" containerName="maas-api" containerID="cri-o://f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3" gracePeriod=30
Jul 07 18:58:26.305377 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.305350    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:58:26.343056 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.343027    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvpml\" (UniqueName: \"kubernetes.io/projected/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-kube-api-access-vvpml\") pod \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") "
Jul 07 18:58:26.343229 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.343067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-maas-api-tls\") pod \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") "
Jul 07 18:58:26.343229 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.343149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-openshift-service-ca\") pod \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\" (UID: \"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8\") "
Jul 07 18:58:26.343526 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.343492    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" (UID: "24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:58:26.345084 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.345061    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-kube-api-access-vvpml" (OuterVolumeSpecName: "kube-api-access-vvpml") pod "24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" (UID: "24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8"). InnerVolumeSpecName "kube-api-access-vvpml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:58:26.345178 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.345128    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" (UID: "24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:58:26.436346 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.436310    2565 generic.go:356] "Generic (PLEG): container finished" podID="24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" containerID="f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3" exitCode=0
Jul 07 18:58:26.436529 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.436352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"
Jul 07 18:58:26.436529 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.436363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq" event={"ID":"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8","Type":"ContainerDied","Data":"f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3"}
Jul 07 18:58:26.436529 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.436409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq" event={"ID":"24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8","Type":"ContainerDied","Data":"6a5b11e38e1f635cab8ed1a9b5c9492a456e4df5472d07fc6ce320a52a3c5dce"}
Jul 07 18:58:26.436529 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.436433    2565 scope.go:117] "RemoveContainer" containerID="f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3"
Jul 07 18:58:26.443962 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.443873    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:58:26.443962 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.443896    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvpml\" (UniqueName: \"kubernetes.io/projected/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-kube-api-access-vvpml\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:58:26.443962 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.443911    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:58:26.445390 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.445373    2565 scope.go:117] "RemoveContainer" containerID="f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3"
Jul 07 18:58:26.445668 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:58:26.445645    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3\": container with ID starting with f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3 not found: ID does not exist" containerID="f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3"
Jul 07 18:58:26.445724 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.445681    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3"} err="failed to get container status \"f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3\": rpc error: code = NotFound desc = could not find container \"f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3\": container with ID starting with f28e826d75e136fe1962e228542c2c91f159f81f95859235be31423a69dd7cb3 not found: ID does not exist"
Jul 07 18:58:26.467766 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.467743    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"]
Jul 07 18:58:26.486161 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:26.486135    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c5164d-595f85b4df-7dkjq"]
Jul 07 18:58:27.048468 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:27.048435    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" path="/var/lib/kubelet/pods/24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8/volumes"
Jul 07 18:58:37.724415 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.724374    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"]
Jul 07 18:58:37.725280 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725257    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" containerName="maas-api"
Jul 07 18:58:37.725375 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725283    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" containerName="maas-api"
Jul 07 18:58:37.725375 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725294    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13cf7d72-4356-42a3-8e11-d618ce117798" containerName="main"
Jul 07 18:58:37.725375 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725303    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13cf7d72-4356-42a3-8e11-d618ce117798" containerName="main"
Jul 07 18:58:37.725375 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725324    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" containerName="maas-api"
Jul 07 18:58:37.725375 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725333    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" containerName="maas-api"
Jul 07 18:58:37.725624 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725538    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81f3c24-e6ea-4c20-91eb-06b52b5b6a8e" containerName="maas-api"
Jul 07 18:58:37.725624 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725561    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13cf7d72-4356-42a3-8e11-d618ce117798" containerName="main"
Jul 07 18:58:37.725624 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.725572    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c5abcd-b0b1-4b80-a199-3d8e9d2d58a8" containerName="maas-api"
Jul 07 18:58:37.729632 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.729353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.734552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.734524    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:58:37.734714 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.734696    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-10b7bf-serving-cert\""
Jul 07 18:58:37.764979 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.764958    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"]
Jul 07 18:58:37.844069 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.844041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2686cb-fc34-4ed5-9641-22367c30c8c0-maas-api-tls\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.844213 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.844078    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2686cb-fc34-4ed5-9641-22367c30c8c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.844213 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.844098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4xg\" (UniqueName: \"kubernetes.io/projected/2b2686cb-fc34-4ed5-9641-22367c30c8c0-kube-api-access-rk4xg\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.945074 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.945049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2686cb-fc34-4ed5-9641-22367c30c8c0-maas-api-tls\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.945219 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.945087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2686cb-fc34-4ed5-9641-22367c30c8c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.945219 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.945105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4xg\" (UniqueName: \"kubernetes.io/projected/2b2686cb-fc34-4ed5-9641-22367c30c8c0-kube-api-access-rk4xg\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.945701 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.945680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2686cb-fc34-4ed5-9641-22367c30c8c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.947430 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.947400    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2686cb-fc34-4ed5-9641-22367c30c8c0-maas-api-tls\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:37.972836 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:37.972815    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4xg\" (UniqueName: \"kubernetes.io/projected/2b2686cb-fc34-4ed5-9641-22367c30c8c0-kube-api-access-rk4xg\") pod \"maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") " pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:38.040596 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:38.040539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:38.184829 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:38.184796    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"]
Jul 07 18:58:38.188893 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:58:38.188868    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2686cb_fc34_4ed5_9641_22367c30c8c0.slice/crio-5e0dada211e8bc474810625cf59dda715ce0e3ad1a3622bfaf7fe671da8c703d WatchSource:0}: Error finding container 5e0dada211e8bc474810625cf59dda715ce0e3ad1a3622bfaf7fe671da8c703d: Status 404 returned error can't find the container with id 5e0dada211e8bc474810625cf59dda715ce0e3ad1a3622bfaf7fe671da8c703d
Jul 07 18:58:38.492412 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:38.492379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh" event={"ID":"2b2686cb-fc34-4ed5-9641-22367c30c8c0","Type":"ContainerStarted","Data":"5e0dada211e8bc474810625cf59dda715ce0e3ad1a3622bfaf7fe671da8c703d"}
Jul 07 18:58:39.499037 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:39.498995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh" event={"ID":"2b2686cb-fc34-4ed5-9641-22367c30c8c0","Type":"ContainerStarted","Data":"3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52"}
Jul 07 18:58:39.499559 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:39.499065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:39.632097 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:39.632044    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh" podStartSLOduration=2.218109534 podStartE2EDuration="2.632027413s" totalImagesPullingTime="413.917879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:58:39.592862816 +0000 UTC m=+2147.168626709" watchObservedRunningTime="2026-07-07 18:58:39.632027413 +0000 UTC m=+2147.207791306"
Jul 07 18:58:45.510184 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:45.510095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:58:46.257647 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.257615    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"]
Jul 07 18:58:46.261436 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.261415    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.284037 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.284012    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-90d474-serving-cert\""
Jul 07 18:58:46.285957 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.285934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"]
Jul 07 18:58:46.315626 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.315601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.315768 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.315639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx7n5\" (UniqueName: \"kubernetes.io/projected/6bd1a21b-f058-4b39-b1b5-327345a0066f-kube-api-access-xx7n5\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.315768 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.315684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd1a21b-f058-4b39-b1b5-327345a0066f-openshift-service-ca\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.416713 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.416684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.416853 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.416722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx7n5\" (UniqueName: \"kubernetes.io/projected/6bd1a21b-f058-4b39-b1b5-327345a0066f-kube-api-access-xx7n5\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.416853 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.416769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd1a21b-f058-4b39-b1b5-327345a0066f-openshift-service-ca\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.416853 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:58:46.416830    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-90d474-serving-cert: secret "maas-api-e2e-api-b-90d474-serving-cert" not found
Jul 07 18:58:46.417047 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:58:46.416884    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls podName:6bd1a21b-f058-4b39-b1b5-327345a0066f nodeName:}" failed. No retries permitted until 2026-07-07 18:58:46.91686679 +0000 UTC m=+2154.492630667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls") pod "maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" (UID: "6bd1a21b-f058-4b39-b1b5-327345a0066f") : secret "maas-api-e2e-api-b-90d474-serving-cert" not found
Jul 07 18:58:46.417385 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.417366    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd1a21b-f058-4b39-b1b5-327345a0066f-openshift-service-ca\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.442845 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.442820    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx7n5\" (UniqueName: \"kubernetes.io/projected/6bd1a21b-f058-4b39-b1b5-327345a0066f-kube-api-access-xx7n5\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.920604 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.920561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:46.922966 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:46.922940    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls\") pod \"maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") " pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:47.173306 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:47.173213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:47.341353 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:47.341326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"]
Jul 07 18:58:47.343074 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:58:47.343050    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd1a21b_f058_4b39_b1b5_327345a0066f.slice/crio-20f69d71504de851870c23168b9f5c239b459dd45d0b6d5567623caf57f02863 WatchSource:0}: Error finding container 20f69d71504de851870c23168b9f5c239b459dd45d0b6d5567623caf57f02863: Status 404 returned error can't find the container with id 20f69d71504de851870c23168b9f5c239b459dd45d0b6d5567623caf57f02863
Jul 07 18:58:47.538329 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:47.538243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" event={"ID":"6bd1a21b-f058-4b39-b1b5-327345a0066f","Type":"ContainerStarted","Data":"20f69d71504de851870c23168b9f5c239b459dd45d0b6d5567623caf57f02863"}
Jul 07 18:58:48.544149 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:48.544110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" event={"ID":"6bd1a21b-f058-4b39-b1b5-327345a0066f","Type":"ContainerStarted","Data":"63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b"}
Jul 07 18:58:48.544558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:48.544233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:58:48.578950 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:48.578905    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" podStartSLOduration=2.180818056 podStartE2EDuration="2.578893347s" totalImagesPullingTime="398.075291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:58:48.577244325 +0000 UTC m=+2156.153008237" watchObservedRunningTime="2026-07-07 18:58:48.578893347 +0000 UTC m=+2156.154657267"
Jul 07 18:58:54.554372 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:58:54.554339    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:59:00.202325 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.202295    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"]
Jul 07 18:59:00.202739 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.202503    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh" podUID="2b2686cb-fc34-4ed5-9641-22367c30c8c0" containerName="maas-api" containerID="cri-o://3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52" gracePeriod=30
Jul 07 18:59:00.445814 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.445790    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:59:00.544276 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.544203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk4xg\" (UniqueName: \"kubernetes.io/projected/2b2686cb-fc34-4ed5-9641-22367c30c8c0-kube-api-access-rk4xg\") pod \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") "
Jul 07 18:59:00.544276 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.544252    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2686cb-fc34-4ed5-9641-22367c30c8c0-maas-api-tls\") pod \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") "
Jul 07 18:59:00.544519 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.544355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2686cb-fc34-4ed5-9641-22367c30c8c0-openshift-service-ca\") pod \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\" (UID: \"2b2686cb-fc34-4ed5-9641-22367c30c8c0\") "
Jul 07 18:59:00.544725 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.544706    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2686cb-fc34-4ed5-9641-22367c30c8c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b2686cb-fc34-4ed5-9641-22367c30c8c0" (UID: "2b2686cb-fc34-4ed5-9641-22367c30c8c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:59:00.546318 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.546299    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2686cb-fc34-4ed5-9641-22367c30c8c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b2686cb-fc34-4ed5-9641-22367c30c8c0" (UID: "2b2686cb-fc34-4ed5-9641-22367c30c8c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:59:00.546411 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.546320    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2686cb-fc34-4ed5-9641-22367c30c8c0-kube-api-access-rk4xg" (OuterVolumeSpecName: "kube-api-access-rk4xg") pod "2b2686cb-fc34-4ed5-9641-22367c30c8c0" (UID: "2b2686cb-fc34-4ed5-9641-22367c30c8c0"). InnerVolumeSpecName "kube-api-access-rk4xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:59:00.602224 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.602199    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b2686cb-fc34-4ed5-9641-22367c30c8c0" containerID="3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52" exitCode=0
Jul 07 18:59:00.602319 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.602243    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"
Jul 07 18:59:00.602319 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.602287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh" event={"ID":"2b2686cb-fc34-4ed5-9641-22367c30c8c0","Type":"ContainerDied","Data":"3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52"}
Jul 07 18:59:00.602407 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.602341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh" event={"ID":"2b2686cb-fc34-4ed5-9641-22367c30c8c0","Type":"ContainerDied","Data":"5e0dada211e8bc474810625cf59dda715ce0e3ad1a3622bfaf7fe671da8c703d"}
Jul 07 18:59:00.602407 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.602371    2565 scope.go:117] "RemoveContainer" containerID="3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52"
Jul 07 18:59:00.610698 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.610683    2565 scope.go:117] "RemoveContainer" containerID="3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52"
Jul 07 18:59:00.610947 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:59:00.610926    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52\": container with ID starting with 3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52 not found: ID does not exist" containerID="3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52"
Jul 07 18:59:00.610996 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.610958    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52"} err="failed to get container status \"3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52\": rpc error: code = NotFound desc = could not find container \"3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52\": container with ID starting with 3b0f648db0b13c09225a935da2129199fd146b374871ef197f8e31637f2f7f52 not found: ID does not exist"
Jul 07 18:59:00.625558 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.625538    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"]
Jul 07 18:59:00.637880 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.637860    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-10b7bf-7bb7c9fc8d-vstgh"]
Jul 07 18:59:00.645598 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.645581    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4xg\" (UniqueName: \"kubernetes.io/projected/2b2686cb-fc34-4ed5-9641-22367c30c8c0-kube-api-access-rk4xg\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:59:00.645673 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.645600    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2686cb-fc34-4ed5-9641-22367c30c8c0-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:59:00.645673 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:00.645610    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2686cb-fc34-4ed5-9641-22367c30c8c0-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:59:01.048378 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:01.048346    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2686cb-fc34-4ed5-9641-22367c30c8c0" path="/var/lib/kubelet/pods/2b2686cb-fc34-4ed5-9641-22367c30c8c0/volumes"
Jul 07 18:59:08.798161 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:08.798127    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"]
Jul 07 18:59:08.798660 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:08.798338    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" podUID="6bd1a21b-f058-4b39-b1b5-327345a0066f" containerName="maas-api" containerID="cri-o://63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b" gracePeriod=30
Jul 07 18:59:09.045498 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.045475    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:59:09.215897 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.215872    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls\") pod \"6bd1a21b-f058-4b39-b1b5-327345a0066f\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") "
Jul 07 18:59:09.216046 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.215904    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx7n5\" (UniqueName: \"kubernetes.io/projected/6bd1a21b-f058-4b39-b1b5-327345a0066f-kube-api-access-xx7n5\") pod \"6bd1a21b-f058-4b39-b1b5-327345a0066f\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") "
Jul 07 18:59:09.216046 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.215953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd1a21b-f058-4b39-b1b5-327345a0066f-openshift-service-ca\") pod \"6bd1a21b-f058-4b39-b1b5-327345a0066f\" (UID: \"6bd1a21b-f058-4b39-b1b5-327345a0066f\") "
Jul 07 18:59:09.216302 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.216278    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd1a21b-f058-4b39-b1b5-327345a0066f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bd1a21b-f058-4b39-b1b5-327345a0066f" (UID: "6bd1a21b-f058-4b39-b1b5-327345a0066f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:59:09.217955 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.217929    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bd1a21b-f058-4b39-b1b5-327345a0066f" (UID: "6bd1a21b-f058-4b39-b1b5-327345a0066f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:59:09.218037 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.218003    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd1a21b-f058-4b39-b1b5-327345a0066f-kube-api-access-xx7n5" (OuterVolumeSpecName: "kube-api-access-xx7n5") pod "6bd1a21b-f058-4b39-b1b5-327345a0066f" (UID: "6bd1a21b-f058-4b39-b1b5-327345a0066f"). InnerVolumeSpecName "kube-api-access-xx7n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:59:09.317258 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.317232    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd1a21b-f058-4b39-b1b5-327345a0066f-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:59:09.317258 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.317256    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx7n5\" (UniqueName: \"kubernetes.io/projected/6bd1a21b-f058-4b39-b1b5-327345a0066f-kube-api-access-xx7n5\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:59:09.317388 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.317266    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd1a21b-f058-4b39-b1b5-327345a0066f-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 18:59:09.645213 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.645189    2565 generic.go:356] "Generic (PLEG): container finished" podID="6bd1a21b-f058-4b39-b1b5-327345a0066f" containerID="63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b" exitCode=0
Jul 07 18:59:09.645330 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.645233    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"
Jul 07 18:59:09.645330 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.645260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" event={"ID":"6bd1a21b-f058-4b39-b1b5-327345a0066f","Type":"ContainerDied","Data":"63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b"}
Jul 07 18:59:09.645330 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.645298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn" event={"ID":"6bd1a21b-f058-4b39-b1b5-327345a0066f","Type":"ContainerDied","Data":"20f69d71504de851870c23168b9f5c239b459dd45d0b6d5567623caf57f02863"}
Jul 07 18:59:09.645330 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.645319    2565 scope.go:117] "RemoveContainer" containerID="63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b"
Jul 07 18:59:09.654136 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.654118    2565 scope.go:117] "RemoveContainer" containerID="63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b"
Jul 07 18:59:09.654376 ip-10-0-130-31 kubenswrapper[2565]: E0707 18:59:09.654358    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b\": container with ID starting with 63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b not found: ID does not exist" containerID="63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b"
Jul 07 18:59:09.654422 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.654386    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b"} err="failed to get container status \"63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b\": rpc error: code = NotFound desc = could not find container \"63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b\": container with ID starting with 63409dd6e6e7310e54bae6de712a7cabe8ffd2b0a30a3209243a2271e28cf21b not found: ID does not exist"
Jul 07 18:59:09.674701 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.674677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"]
Jul 07 18:59:09.681904 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:09.681885    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-90d474-7c69458dbf-jnnmn"]
Jul 07 18:59:11.047109 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:11.047080    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd1a21b-f058-4b39-b1b5-327345a0066f" path="/var/lib/kubelet/pods/6bd1a21b-f058-4b39-b1b5-327345a0066f/volumes"
Jul 07 18:59:19.293522 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293489    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"]
Jul 07 18:59:19.293965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293798    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd1a21b-f058-4b39-b1b5-327345a0066f" containerName="maas-api"
Jul 07 18:59:19.293965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293809    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd1a21b-f058-4b39-b1b5-327345a0066f" containerName="maas-api"
Jul 07 18:59:19.293965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293824    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2686cb-fc34-4ed5-9641-22367c30c8c0" containerName="maas-api"
Jul 07 18:59:19.293965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293830    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2686cb-fc34-4ed5-9641-22367c30c8c0" containerName="maas-api"
Jul 07 18:59:19.293965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293895    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd1a21b-f058-4b39-b1b5-327345a0066f" containerName="maas-api"
Jul 07 18:59:19.293965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.293903    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2686cb-fc34-4ed5-9641-22367c30c8c0" containerName="maas-api"
Jul 07 18:59:19.298096 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.298076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.322819 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.322795    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-972159-serving-cert\""
Jul 07 18:59:19.322929 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.322801    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-4v97g\""
Jul 07 18:59:19.356796 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.356770    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"]
Jul 07 18:59:19.392568 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.392541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51655696-3760-4af5-94bc-5d6496574424-maas-api-tls\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.392660 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.392595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51655696-3760-4af5-94bc-5d6496574424-openshift-service-ca\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.392715 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.392662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xctc7\" (UniqueName: \"kubernetes.io/projected/51655696-3760-4af5-94bc-5d6496574424-kube-api-access-xctc7\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.493506 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.493480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51655696-3760-4af5-94bc-5d6496574424-maas-api-tls\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.493609 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.493534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51655696-3760-4af5-94bc-5d6496574424-openshift-service-ca\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.493609 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.493569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xctc7\" (UniqueName: \"kubernetes.io/projected/51655696-3760-4af5-94bc-5d6496574424-kube-api-access-xctc7\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.494205 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.494183    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51655696-3760-4af5-94bc-5d6496574424-openshift-service-ca\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.495942 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.495922    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51655696-3760-4af5-94bc-5d6496574424-maas-api-tls\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.509938 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.509912    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xctc7\" (UniqueName: \"kubernetes.io/projected/51655696-3760-4af5-94bc-5d6496574424-kube-api-access-xctc7\") pod \"maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") " pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.607732 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.607707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:19.755264 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:19.755238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"]
Jul 07 18:59:19.757477 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:59:19.757438    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51655696_3760_4af5_94bc_5d6496574424.slice/crio-bd3f2bbc4925ca5e2540056960e33c61b74ddf1ac154bfe2d066799656c0a742 WatchSource:0}: Error finding container bd3f2bbc4925ca5e2540056960e33c61b74ddf1ac154bfe2d066799656c0a742: Status 404 returned error can't find the container with id bd3f2bbc4925ca5e2540056960e33c61b74ddf1ac154bfe2d066799656c0a742
Jul 07 18:59:20.705958 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:20.705919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9" event={"ID":"51655696-3760-4af5-94bc-5d6496574424","Type":"ContainerStarted","Data":"272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75"}
Jul 07 18:59:20.706347 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:20.705986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9" event={"ID":"51655696-3760-4af5-94bc-5d6496574424","Type":"ContainerStarted","Data":"bd3f2bbc4925ca5e2540056960e33c61b74ddf1ac154bfe2d066799656c0a742"}
Jul 07 18:59:20.706347 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:20.706108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:20.731917 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:20.731873    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9" podStartSLOduration=1.398727468 podStartE2EDuration="1.731859321s" totalImagesPullingTime="333.131853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:59:20.729355127 +0000 UTC m=+2188.305119021" watchObservedRunningTime="2026-07-07 18:59:20.731859321 +0000 UTC m=+2188.307623214"
Jul 07 18:59:26.715965 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:26.715935    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 18:59:27.318362 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.318326    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"]
Jul 07 18:59:27.322440 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.322420    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.324975 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.324955    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-e7285b-serving-cert\""
Jul 07 18:59:27.345297 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.345274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"]
Jul 07 18:59:27.356449 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.356426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2915b186-f751-4926-a689-1579ea1ab147-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.356552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.356472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915b186-f751-4926-a689-1579ea1ab147-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.356552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.356524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/2915b186-f751-4926-a689-1579ea1ab147-kube-api-access-r5qn7\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.457474 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.457427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2915b186-f751-4926-a689-1579ea1ab147-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.457608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.457481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915b186-f751-4926-a689-1579ea1ab147-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.457608 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.457511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/2915b186-f751-4926-a689-1579ea1ab147-kube-api-access-r5qn7\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.458247 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.458231    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915b186-f751-4926-a689-1579ea1ab147-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.459881 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.459864    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2915b186-f751-4926-a689-1579ea1ab147-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.474531 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.474509    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/2915b186-f751-4926-a689-1579ea1ab147-kube-api-access-r5qn7\") pod \"maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") " pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.635269 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.635244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:27.771485 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:27.771430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"]
Jul 07 18:59:27.774836 ip-10-0-130-31 kubenswrapper[2565]: W0707 18:59:27.774804    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2915b186_f751_4926_a689_1579ea1ab147.slice/crio-5151c487d25e759f82e51cb43b6537bb4a3ced80b7051485c1b062245d856749 WatchSource:0}: Error finding container 5151c487d25e759f82e51cb43b6537bb4a3ced80b7051485c1b062245d856749: Status 404 returned error can't find the container with id 5151c487d25e759f82e51cb43b6537bb4a3ced80b7051485c1b062245d856749
Jul 07 18:59:28.744386 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:28.744349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t" event={"ID":"2915b186-f751-4926-a689-1579ea1ab147","Type":"ContainerStarted","Data":"afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b"}
Jul 07 18:59:28.744552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:28.744410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t" event={"ID":"2915b186-f751-4926-a689-1579ea1ab147","Type":"ContainerStarted","Data":"5151c487d25e759f82e51cb43b6537bb4a3ced80b7051485c1b062245d856749"}
Jul 07 18:59:28.744552 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:28.744450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 18:59:28.770694 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:28.770648    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t" podStartSLOduration=1.413714772 podStartE2EDuration="1.770634546s" totalImagesPullingTime="356.919774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:59:28.769286297 +0000 UTC m=+2196.345050216" watchObservedRunningTime="2026-07-07 18:59:28.770634546 +0000 UTC m=+2196.346398439"
Jul 07 18:59:34.754946 ip-10-0-130-31 kubenswrapper[2565]: I0707 18:59:34.754916    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 19:02:21.635615 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.635572    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"]
Jul 07 19:02:21.639398 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.639375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.641927 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.641909    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"kube-root-ca.crt\""
Jul 07 19:02:21.642818 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.642795    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"sub-test-model-e7285b-kserve-self-signed-certs\""
Jul 07 19:02:21.642917 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.642871    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"openshift-service-ca.crt\""
Jul 07 19:02:21.642917 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.642893    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"default-dockercfg-5t9b6\""
Jul 07 19:02:21.648132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.648110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"]
Jul 07 19:02:21.792953 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.792910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7vs\" (UniqueName: \"kubernetes.io/projected/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-kube-api-access-mj7vs\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.793132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.792968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-home\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.793132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.793040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tmp-dir\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.793132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.793069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tls-certs\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.793132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.793089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-model-cache\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.793274 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.793176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-dshm\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-model-cache\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894132 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-dshm\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894334 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7vs\" (UniqueName: \"kubernetes.io/projected/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-kube-api-access-mj7vs\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894334 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-home\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894334 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tmp-dir\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894334 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tls-certs\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894595 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894512    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-model-cache\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894665 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894646    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tmp-dir\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.894721 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.894664    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-home\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.896487 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.896451    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-dshm\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.896760 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.896742    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tls-certs\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.903469 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.903431    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7vs\" (UniqueName: \"kubernetes.io/projected/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-kube-api-access-mj7vs\") pod \"sub-test-model-e7285b-kserve-5cdb758d89-5ndgw\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") " pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:21.948981 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:21.948946    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:22.085773 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:22.085745    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"]
Jul 07 19:02:22.087468 ip-10-0-130-31 kubenswrapper[2565]: W0707 19:02:22.087425    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea8e76e9_4981_4a00_89bb_b6b9d2dcfd39.slice/crio-b4f3ab5117a6c4c156423ed12fe15ab884c7775a97283e371e4801abad5185ff WatchSource:0}: Error finding container b4f3ab5117a6c4c156423ed12fe15ab884c7775a97283e371e4801abad5185ff: Status 404 returned error can't find the container with id b4f3ab5117a6c4c156423ed12fe15ab884c7775a97283e371e4801abad5185ff
Jul 07 19:02:22.539731 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:22.539632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw" event={"ID":"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39","Type":"ContainerStarted","Data":"b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489"}
Jul 07 19:02:22.539731 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:22.539683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw" event={"ID":"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39","Type":"ContainerStarted","Data":"b4f3ab5117a6c4c156423ed12fe15ab884c7775a97283e371e4801abad5185ff"}
Jul 07 19:02:22.559171 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:22.559119    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw" podStartSLOduration=1.559102594 podStartE2EDuration="1.559102594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:22.557626255 +0000 UTC m=+2370.133390166" watchObservedRunningTime="2026-07-07 19:02:22.559102594 +0000 UTC m=+2370.134866488"
Jul 07 19:02:31.949477 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:31.949425    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:31.950035 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:31.949500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:31.961719 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:31.961687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:02:32.595598 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:02:32.595569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:03:14.072885 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:14.072846    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"]
Jul 07 19:03:29.092814 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.092767    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw" podUID="ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" containerName="main" containerID="cri-o://b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489" gracePeriod=45
Jul 07 19:03:29.340319 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.340289    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:03:29.405653 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj7vs\" (UniqueName: \"kubernetes.io/projected/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-kube-api-access-mj7vs\") pod \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") "
Jul 07 19:03:29.405806 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405677    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-home\") pod \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") "
Jul 07 19:03:29.405806 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tmp-dir\") pod \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") "
Jul 07 19:03:29.405806 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-model-cache\") pod \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") "
Jul 07 19:03:29.405806 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-dshm\") pod \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") "
Jul 07 19:03:29.405806 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tls-certs\") pod \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\" (UID: \"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39\") "
Jul 07 19:03:29.406078 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405969    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" (UID: "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:03:29.406078 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.405989    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" (UID: "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:03:29.406078 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.406016    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-home" (OuterVolumeSpecName: "home") pod "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" (UID: "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:03:29.406237 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.406144    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-home\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:03:29.406237 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.406162    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tmp-dir\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:03:29.406237 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.406175    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-model-cache\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:03:29.407889 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.407863    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-kube-api-access-mj7vs" (OuterVolumeSpecName: "kube-api-access-mj7vs") pod "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" (UID: "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39"). InnerVolumeSpecName "kube-api-access-mj7vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:03:29.408015 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.407956    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-dshm" (OuterVolumeSpecName: "dshm") pod "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" (UID: "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:03:29.408063 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.408008    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" (UID: "ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:03:29.507875 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.507834    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-dshm\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:03:29.507875 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.507875    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-tls-certs\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:03:29.508077 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.507893    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7vs\" (UniqueName: \"kubernetes.io/projected/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39-kube-api-access-mj7vs\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:03:29.841516 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.841455    2565 generic.go:356] "Generic (PLEG): container finished" podID="ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" containerID="b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489" exitCode=0
Jul 07 19:03:29.841516 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.841517    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"
Jul 07 19:03:29.841796 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.841560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw" event={"ID":"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39","Type":"ContainerDied","Data":"b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489"}
Jul 07 19:03:29.841796 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.841606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw" event={"ID":"ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39","Type":"ContainerDied","Data":"b4f3ab5117a6c4c156423ed12fe15ab884c7775a97283e371e4801abad5185ff"}
Jul 07 19:03:29.841796 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.841625    2565 scope.go:117] "RemoveContainer" containerID="b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489"
Jul 07 19:03:29.850693 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.850668    2565 scope.go:117] "RemoveContainer" containerID="b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489"
Jul 07 19:03:29.850975 ip-10-0-130-31 kubenswrapper[2565]: E0707 19:03:29.850956    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489\": container with ID starting with b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489 not found: ID does not exist" containerID="b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489"
Jul 07 19:03:29.851046 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.850984    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489"} err="failed to get container status \"b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489\": rpc error: code = NotFound desc = could not find container \"b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489\": container with ID starting with b5f9e5d167a3f24700aa0116ec6aad520b1eff3c76c7c79de445ce9019dda489 not found: ID does not exist"
Jul 07 19:03:29.863165 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.863142    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"]
Jul 07 19:03:29.866613 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:29.866590    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/sub-test-model-e7285b-kserve-5cdb758d89-5ndgw"]
Jul 07 19:03:31.047217 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:31.047188    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" path="/var/lib/kubelet/pods/ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39/volumes"
Jul 07 19:03:56.551691 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.551650    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"]
Jul 07 19:03:56.552091 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.552016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" containerName="main"
Jul 07 19:03:56.552091 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.552026    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" containerName="main"
Jul 07 19:03:56.552170 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.552095    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea8e76e9-4981-4a00-89bb-b6b9d2dcfd39" containerName="main"
Jul 07 19:03:56.556643 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.556623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.560968 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.560948    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"default-dockercfg-5t9b6\""
Jul 07 19:03:56.561779 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.561763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"kube-root-ca.crt\""
Jul 07 19:03:56.561869 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.561842    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"openshift-service-ca.crt\""
Jul 07 19:03:56.561925 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.561842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"rate-test-model-e7285b-kserve-self-signed-certs\""
Jul 07 19:03:56.567729 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.567710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"]
Jul 07 19:03:56.655792 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.655759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tls-certs\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.655792 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.655795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pkb\" (UniqueName: \"kubernetes.io/projected/43f71c8c-32f6-4984-a5f6-08d0edc0a742-kube-api-access-55pkb\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.656026 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.655897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tmp-dir\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.656026 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.655971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-home\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.656026 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.656013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-model-cache\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.656138 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.656043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-dshm\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757077 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-dshm\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757077 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tls-certs\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757280 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pkb\" (UniqueName: \"kubernetes.io/projected/43f71c8c-32f6-4984-a5f6-08d0edc0a742-kube-api-access-55pkb\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757280 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tmp-dir\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757280 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-home\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757280 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-model-cache\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757589 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757570    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tmp-dir\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757664 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757628    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-model-cache\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.757725 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.757675    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-home\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.759356 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.759338    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-dshm\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.759695 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.759674    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tls-certs\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.767889 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.767830    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pkb\" (UniqueName: \"kubernetes.io/projected/43f71c8c-32f6-4984-a5f6-08d0edc0a742-kube-api-access-55pkb\") pod \"rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") " pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:56.866235 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:56.866209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:03:57.008063 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:57.008039    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"]
Jul 07 19:03:57.010597 ip-10-0-130-31 kubenswrapper[2565]: W0707 19:03:57.010568    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43f71c8c_32f6_4984_a5f6_08d0edc0a742.slice/crio-e6a705adbb30c91c8a903c186e0bc47ecee47f62c4eb96b02a76b1fa8474ec9d WatchSource:0}: Error finding container e6a705adbb30c91c8a903c186e0bc47ecee47f62c4eb96b02a76b1fa8474ec9d: Status 404 returned error can't find the container with id e6a705adbb30c91c8a903c186e0bc47ecee47f62c4eb96b02a76b1fa8474ec9d
Jul 07 19:03:57.974570 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:57.974536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5" event={"ID":"43f71c8c-32f6-4984-a5f6-08d0edc0a742","Type":"ContainerStarted","Data":"ce7a5a4e8d180ecd92db072c45092b32159f179c415a212a5a7053d0390a9c5e"}
Jul 07 19:03:57.974958 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:57.974580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5" event={"ID":"43f71c8c-32f6-4984-a5f6-08d0edc0a742","Type":"ContainerStarted","Data":"e6a705adbb30c91c8a903c186e0bc47ecee47f62c4eb96b02a76b1fa8474ec9d"}
Jul 07 19:03:57.997668 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:03:57.997620    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5" podStartSLOduration=1.997603582 podStartE2EDuration="1.997603582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:03:57.996241806 +0000 UTC m=+2465.572005701" watchObservedRunningTime="2026-07-07 19:03:57.997603582 +0000 UTC m=+2465.573367476"
Jul 07 19:04:06.866944 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:06.866905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:04:06.866944 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:06.866956    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:04:06.879808 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:06.879780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:04:07.032792 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:07.032765    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:04:42.053946 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:42.053913    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"]
Jul 07 19:04:43.938609 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:43.938573    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"]
Jul 07 19:04:43.939010 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:43.938775    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9" podUID="51655696-3760-4af5-94bc-5d6496574424" containerName="maas-api" containerID="cri-o://272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75" gracePeriod=30
Jul 07 19:04:44.182015 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.181992    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 19:04:44.193765 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.193690    2565 generic.go:356] "Generic (PLEG): container finished" podID="51655696-3760-4af5-94bc-5d6496574424" containerID="272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75" exitCode=0
Jul 07 19:04:44.193765 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.193731    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"
Jul 07 19:04:44.193924 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.193772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9" event={"ID":"51655696-3760-4af5-94bc-5d6496574424","Type":"ContainerDied","Data":"272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75"}
Jul 07 19:04:44.193924 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.193820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9" event={"ID":"51655696-3760-4af5-94bc-5d6496574424","Type":"ContainerDied","Data":"bd3f2bbc4925ca5e2540056960e33c61b74ddf1ac154bfe2d066799656c0a742"}
Jul 07 19:04:44.193924 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.193842    2565 scope.go:117] "RemoveContainer" containerID="272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75"
Jul 07 19:04:44.203920 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.203893    2565 scope.go:117] "RemoveContainer" containerID="272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75"
Jul 07 19:04:44.204219 ip-10-0-130-31 kubenswrapper[2565]: E0707 19:04:44.204196    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75\": container with ID starting with 272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75 not found: ID does not exist" containerID="272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75"
Jul 07 19:04:44.204314 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.204229    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75"} err="failed to get container status \"272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75\": rpc error: code = NotFound desc = could not find container \"272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75\": container with ID starting with 272a83cfcceeef647417108421add92b043ba7cc519e4d8113a280b6781e1c75 not found: ID does not exist"
Jul 07 19:04:44.296730 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.296700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51655696-3760-4af5-94bc-5d6496574424-maas-api-tls\") pod \"51655696-3760-4af5-94bc-5d6496574424\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") "
Jul 07 19:04:44.296882 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.296798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51655696-3760-4af5-94bc-5d6496574424-openshift-service-ca\") pod \"51655696-3760-4af5-94bc-5d6496574424\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") "
Jul 07 19:04:44.296882 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.296842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xctc7\" (UniqueName: \"kubernetes.io/projected/51655696-3760-4af5-94bc-5d6496574424-kube-api-access-xctc7\") pod \"51655696-3760-4af5-94bc-5d6496574424\" (UID: \"51655696-3760-4af5-94bc-5d6496574424\") "
Jul 07 19:04:44.297149 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.297123    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51655696-3760-4af5-94bc-5d6496574424-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51655696-3760-4af5-94bc-5d6496574424" (UID: "51655696-3760-4af5-94bc-5d6496574424"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:04:44.298801 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.298781    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51655696-3760-4af5-94bc-5d6496574424-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51655696-3760-4af5-94bc-5d6496574424" (UID: "51655696-3760-4af5-94bc-5d6496574424"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:04:44.298977 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.298955    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51655696-3760-4af5-94bc-5d6496574424-kube-api-access-xctc7" (OuterVolumeSpecName: "kube-api-access-xctc7") pod "51655696-3760-4af5-94bc-5d6496574424" (UID: "51655696-3760-4af5-94bc-5d6496574424"). InnerVolumeSpecName "kube-api-access-xctc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:04:44.398084 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.398053    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51655696-3760-4af5-94bc-5d6496574424-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:44.398084 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.398079    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51655696-3760-4af5-94bc-5d6496574424-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:44.398084 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.398090    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xctc7\" (UniqueName: \"kubernetes.io/projected/51655696-3760-4af5-94bc-5d6496574424-kube-api-access-xctc7\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:44.527054 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.527019    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"]
Jul 07 19:04:44.531530 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:44.531502    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-972159-7dc6f6d864-bk2w9"]
Jul 07 19:04:45.047820 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:45.047788    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51655696-3760-4af5-94bc-5d6496574424" path="/var/lib/kubelet/pods/51655696-3760-4af5-94bc-5d6496574424/volumes"
Jul 07 19:04:57.073391 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.073338    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5" podUID="43f71c8c-32f6-4984-a5f6-08d0edc0a742" containerName="main" containerID="cri-o://ce7a5a4e8d180ecd92db072c45092b32159f179c415a212a5a7053d0390a9c5e" gracePeriod=45
Jul 07 19:04:57.256426 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.256396    2565 generic.go:356] "Generic (PLEG): container finished" podID="43f71c8c-32f6-4984-a5f6-08d0edc0a742" containerID="ce7a5a4e8d180ecd92db072c45092b32159f179c415a212a5a7053d0390a9c5e" exitCode=0
Jul 07 19:04:57.256601 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.256494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5" event={"ID":"43f71c8c-32f6-4984-a5f6-08d0edc0a742","Type":"ContainerDied","Data":"ce7a5a4e8d180ecd92db072c45092b32159f179c415a212a5a7053d0390a9c5e"}
Jul 07 19:04:57.323252 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.323229    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:04:57.415848 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.415772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-dshm\") pod \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") "
Jul 07 19:04:57.415848 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.415814    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55pkb\" (UniqueName: \"kubernetes.io/projected/43f71c8c-32f6-4984-a5f6-08d0edc0a742-kube-api-access-55pkb\") pod \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") "
Jul 07 19:04:57.415848 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.415832    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-home\") pod \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") "
Jul 07 19:04:57.416130 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.415936    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tls-certs\") pod \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") "
Jul 07 19:04:57.416130 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.415960    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tmp-dir\") pod \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") "
Jul 07 19:04:57.416130 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.415991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-model-cache\") pod \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\" (UID: \"43f71c8c-32f6-4984-a5f6-08d0edc0a742\") "
Jul 07 19:04:57.416287 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.416172    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-home" (OuterVolumeSpecName: "home") pod "43f71c8c-32f6-4984-a5f6-08d0edc0a742" (UID: "43f71c8c-32f6-4984-a5f6-08d0edc0a742"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:04:57.416287 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.416220    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43f71c8c-32f6-4984-a5f6-08d0edc0a742" (UID: "43f71c8c-32f6-4984-a5f6-08d0edc0a742"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:04:57.416399 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.416306    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-model-cache" (OuterVolumeSpecName: "model-cache") pod "43f71c8c-32f6-4984-a5f6-08d0edc0a742" (UID: "43f71c8c-32f6-4984-a5f6-08d0edc0a742"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:04:57.416399 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.416365    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-home\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:57.416399 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.416379    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tmp-dir\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:57.417979 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.417954    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-dshm" (OuterVolumeSpecName: "dshm") pod "43f71c8c-32f6-4984-a5f6-08d0edc0a742" (UID: "43f71c8c-32f6-4984-a5f6-08d0edc0a742"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:04:57.418075 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.418033    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f71c8c-32f6-4984-a5f6-08d0edc0a742-kube-api-access-55pkb" (OuterVolumeSpecName: "kube-api-access-55pkb") pod "43f71c8c-32f6-4984-a5f6-08d0edc0a742" (UID: "43f71c8c-32f6-4984-a5f6-08d0edc0a742"). InnerVolumeSpecName "kube-api-access-55pkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:04:57.418075 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.418037    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43f71c8c-32f6-4984-a5f6-08d0edc0a742" (UID: "43f71c8c-32f6-4984-a5f6-08d0edc0a742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:04:57.517690 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.517653    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f71c8c-32f6-4984-a5f6-08d0edc0a742-tls-certs\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:57.517690 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.517685    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-model-cache\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:57.517690 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.517694    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f71c8c-32f6-4984-a5f6-08d0edc0a742-dshm\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:57.517908 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:57.517702    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pkb\" (UniqueName: \"kubernetes.io/projected/43f71c8c-32f6-4984-a5f6-08d0edc0a742-kube-api-access-55pkb\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:04:58.262314 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:58.262280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5" event={"ID":"43f71c8c-32f6-4984-a5f6-08d0edc0a742","Type":"ContainerDied","Data":"e6a705adbb30c91c8a903c186e0bc47ecee47f62c4eb96b02a76b1fa8474ec9d"}
Jul 07 19:04:58.262769 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:58.262297    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"
Jul 07 19:04:58.262769 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:58.262327    2565 scope.go:117] "RemoveContainer" containerID="ce7a5a4e8d180ecd92db072c45092b32159f179c415a212a5a7053d0390a9c5e"
Jul 07 19:04:58.291579 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:58.291552    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"]
Jul 07 19:04:58.296507 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:58.296487    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/rate-test-model-e7285b-kserve-5d8c4f6bff-92lm5"]
Jul 07 19:04:59.047866 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:04:59.047831    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f71c8c-32f6-4984-a5f6-08d0edc0a742" path="/var/lib/kubelet/pods/43f71c8c-32f6-4984-a5f6-08d0edc0a742/volumes"
Jul 07 19:05:27.183990 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.183955    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"]
Jul 07 19:05:27.184787 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.184760    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t" podUID="2915b186-f751-4926-a689-1579ea1ab147" containerName="maas-api" containerID="cri-o://afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b" gracePeriod=30
Jul 07 19:05:27.633608 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.633583    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 19:05:27.795671 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.795579    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/2915b186-f751-4926-a689-1579ea1ab147-kube-api-access-r5qn7\") pod \"2915b186-f751-4926-a689-1579ea1ab147\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") "
Jul 07 19:05:27.795671 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.795615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2915b186-f751-4926-a689-1579ea1ab147-maas-api-tls\") pod \"2915b186-f751-4926-a689-1579ea1ab147\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") "
Jul 07 19:05:27.795671 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.795635    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915b186-f751-4926-a689-1579ea1ab147-openshift-service-ca\") pod \"2915b186-f751-4926-a689-1579ea1ab147\" (UID: \"2915b186-f751-4926-a689-1579ea1ab147\") "
Jul 07 19:05:27.796038 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.796015    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2915b186-f751-4926-a689-1579ea1ab147-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2915b186-f751-4926-a689-1579ea1ab147" (UID: "2915b186-f751-4926-a689-1579ea1ab147"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:05:27.797701 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.797672    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2915b186-f751-4926-a689-1579ea1ab147-kube-api-access-r5qn7" (OuterVolumeSpecName: "kube-api-access-r5qn7") pod "2915b186-f751-4926-a689-1579ea1ab147" (UID: "2915b186-f751-4926-a689-1579ea1ab147"). InnerVolumeSpecName "kube-api-access-r5qn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:05:27.797810 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.797768    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2915b186-f751-4926-a689-1579ea1ab147-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2915b186-f751-4926-a689-1579ea1ab147" (UID: "2915b186-f751-4926-a689-1579ea1ab147"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:05:27.896363 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.896324    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/2915b186-f751-4926-a689-1579ea1ab147-kube-api-access-r5qn7\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:05:27.896363 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.896357    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2915b186-f751-4926-a689-1579ea1ab147-maas-api-tls\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:05:27.896363 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:27.896368    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915b186-f751-4926-a689-1579ea1ab147-openshift-service-ca\") on node \"ip-10-0-130-31.ec2.internal\" DevicePath \"\""
Jul 07 19:05:28.403805 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.403773    2565 generic.go:356] "Generic (PLEG): container finished" podID="2915b186-f751-4926-a689-1579ea1ab147" containerID="afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b" exitCode=0
Jul 07 19:05:28.404283 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.403811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t" event={"ID":"2915b186-f751-4926-a689-1579ea1ab147","Type":"ContainerDied","Data":"afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b"}
Jul 07 19:05:28.404283 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.403851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t" event={"ID":"2915b186-f751-4926-a689-1579ea1ab147","Type":"ContainerDied","Data":"5151c487d25e759f82e51cb43b6537bb4a3ced80b7051485c1b062245d856749"}
Jul 07 19:05:28.404283 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.403871    2565 scope.go:117] "RemoveContainer" containerID="afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b"
Jul 07 19:05:28.404283 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.403813    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"
Jul 07 19:05:28.412777 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.412759    2565 scope.go:117] "RemoveContainer" containerID="afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b"
Jul 07 19:05:28.413026 ip-10-0-130-31 kubenswrapper[2565]: E0707 19:05:28.413007    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b\": container with ID starting with afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b not found: ID does not exist" containerID="afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b"
Jul 07 19:05:28.413087 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.413039    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b"} err="failed to get container status \"afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b\": rpc error: code = NotFound desc = could not find container \"afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b\": container with ID starting with afe8050e450a0c5270ded561b4b54f7a50ef65e10af518d724f0951a7f73501b not found: ID does not exist"
Jul 07 19:05:28.425197 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.425170    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"]
Jul 07 19:05:28.430499 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:28.430479    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-e7285b-64d9769b95-zgs7t"]
Jul 07 19:05:29.047276 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:29.047244    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2915b186-f751-4926-a689-1579ea1ab147" path="/var/lib/kubelet/pods/2915b186-f751-4926-a689-1579ea1ab147/volumes"
Jul 07 19:05:34.949882 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:34.949852    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-789b6bd8c4-v8v98_3d9a5aa9-c9cc-4480-851c-d180c39e5972/authorino/0.log"
Jul 07 19:05:41.044573 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:41.044536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5c6877d755-667k6_a5f32407-ebfc-426f-b950-97b53af468c3/maas-api/0.log"
Jul 07 19:05:41.426979 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:41.426943    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787c79d564-7vvgd_cf8c1666-ca1e-454e-9ba7-1cebfc225725/manager/0.log"
Jul 07 19:05:41.956280 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:41.956249    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-w6rmf_09aefc4d-ab66-40a0-b4f4-41c5ec59e286/copy-manifests/0.log"
Jul 07 19:05:41.963818 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:41.963795    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-w6rmf_09aefc4d-ab66-40a0-b4f4-41c5ec59e286/manager/0.log"
Jul 07 19:05:43.586386 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:43.586358    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-789b6bd8c4-v8v98_3d9a5aa9-c9cc-4480-851c-d180c39e5972/authorino/0.log"
Jul 07 19:05:43.949423 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:43.949336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pwbwp_3595e30a-3255-460a-8e9c-f1b4c14445f8/kuadrant-console-plugin/0.log"
Jul 07 19:05:45.188773 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:45.188742    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-mkbdd_6c84fa22-38ff-4d6e-95f5-8b7a4fec781f/kube-auth-proxy/0.log"
Jul 07 19:05:45.570433 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:45.570410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-z5fsx_2dfa39d3-330f-41aa-85d1-a9abb00ad516/payload-processing/0.log"
Jul 07 19:05:46.056660 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:46.056578    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wm7zg_659f17f0-538a-4884-bde5-4bd83c0b3928/main/0.log"
Jul 07 19:05:46.178546 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:46.178513    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5xtmq_602a30e3-77c9-4efb-85d3-b8d5141789ce/main/0.log"
Jul 07 19:05:46.435411 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:46.435386    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbss4vs_ea5ae3a1-302e-4d61-94bb-1d090abf2896/main/0.log"
Jul 07 19:05:54.753353 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:54.753319    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tvpcl_b7e28baa-d67e-4cb0-b1f4-7f38600ea365/global-pull-secret-syncer/0.log"
Jul 07 19:05:54.863061 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:54.863029    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-982jp_c73f33e5-a952-480a-843a-b3cd9b8ae12a/konnectivity-agent/0.log"
Jul 07 19:05:54.921834 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:05:54.921798    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-31.ec2.internal_bd00b61865beafaa890243361850b718/haproxy/0.log"
Jul 07 19:06:00.172673 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:00.172637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-789b6bd8c4-v8v98_3d9a5aa9-c9cc-4480-851c-d180c39e5972/authorino/0.log"
Jul 07 19:06:00.268391 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:00.268368    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pwbwp_3595e30a-3255-460a-8e9c-f1b4c14445f8/kuadrant-console-plugin/0.log"
Jul 07 19:06:02.611370 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.611338    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-plhh8_14c69045-eaa2-447a-84c6-9ae74eff2017/kube-state-metrics/0.log"
Jul 07 19:06:02.634001 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.633974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-plhh8_14c69045-eaa2-447a-84c6-9ae74eff2017/kube-rbac-proxy-main/0.log"
Jul 07 19:06:02.661145 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.661121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-plhh8_14c69045-eaa2-447a-84c6-9ae74eff2017/kube-rbac-proxy-self/0.log"
Jul 07 19:06:02.724714 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.724637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-f9hcc_54accd2e-5b7f-46a7-b477-9250a42fa77b/monitoring-plugin/0.log"
Jul 07 19:06:02.939812 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.939785    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgqxd_e766d398-d48a-4d6f-915e-e93087dfa0bd/node-exporter/0.log"
Jul 07 19:06:02.968986 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.968953    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgqxd_e766d398-d48a-4d6f-915e-e93087dfa0bd/kube-rbac-proxy/0.log"
Jul 07 19:06:02.993616 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:02.993534    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgqxd_e766d398-d48a-4d6f-915e-e93087dfa0bd/init-textfile/0.log"
Jul 07 19:06:03.126743 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.126707    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/prometheus/0.log"
Jul 07 19:06:03.158498 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.158452    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/config-reloader/0.log"
Jul 07 19:06:03.189570 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.189537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/thanos-sidecar/0.log"
Jul 07 19:06:03.225871 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.225842    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/kube-rbac-proxy-web/0.log"
Jul 07 19:06:03.256632 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.256550    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/kube-rbac-proxy/0.log"
Jul 07 19:06:03.284217 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.284192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/kube-rbac-proxy-thanos/0.log"
Jul 07 19:06:03.319690 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.319668    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94bee8c-19cc-4163-badc-981755cbe606/init-config-reloader/0.log"
Jul 07 19:06:03.361674 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.361647    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-wzd9c_a2fb5bcb-c48d-47cd-b063-88ea579aa5c6/prometheus-operator/0.log"
Jul 07 19:06:03.385330 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.385304    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-wzd9c_a2fb5bcb-c48d-47cd-b063-88ea579aa5c6/kube-rbac-proxy/0.log"
Jul 07 19:06:03.417077 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:03.417050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-gh5v2_7e82a9d5-9214-46f9-a872-e62a8c1daf25/prometheus-operator-admission-webhook/0.log"
Jul 07 19:06:05.013120 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013089    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"]
Jul 07 19:06:05.013524 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013427    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f71c8c-32f6-4984-a5f6-08d0edc0a742" containerName="main"
Jul 07 19:06:05.013524 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013437    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f71c8c-32f6-4984-a5f6-08d0edc0a742" containerName="main"
Jul 07 19:06:05.013524 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51655696-3760-4af5-94bc-5d6496574424" containerName="maas-api"
Jul 07 19:06:05.013524 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013470    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51655696-3760-4af5-94bc-5d6496574424" containerName="maas-api"
Jul 07 19:06:05.013524 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013480    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2915b186-f751-4926-a689-1579ea1ab147" containerName="maas-api"
Jul 07 19:06:05.013524 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2915b186-f751-4926-a689-1579ea1ab147" containerName="maas-api"
Jul 07 19:06:05.013731 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013548    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f71c8c-32f6-4984-a5f6-08d0edc0a742" containerName="main"
Jul 07 19:06:05.013731 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51655696-3760-4af5-94bc-5d6496574424" containerName="maas-api"
Jul 07 19:06:05.013731 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.013565    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2915b186-f751-4926-a689-1579ea1ab147" containerName="maas-api"
Jul 07 19:06:05.016477 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.016447    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.019983 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.019955    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzth9\"/\"openshift-service-ca.crt\""
Jul 07 19:06:05.020100 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.020017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzth9\"/\"kube-root-ca.crt\""
Jul 07 19:06:05.021025 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.021004    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fzth9\"/\"default-dockercfg-fmmcn\""
Jul 07 19:06:05.032094 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.032066    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"]
Jul 07 19:06:05.123448 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.123411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-proc\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.123700 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.123483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xswg\" (UniqueName: \"kubernetes.io/projected/993365b5-7028-4581-b8e7-f653687da7cc-kube-api-access-6xswg\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.123700 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.123513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-podres\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.123700 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.123540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-sys\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.123700 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.123661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-lib-modules\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225131 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-proc\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225131 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xswg\" (UniqueName: \"kubernetes.io/projected/993365b5-7028-4581-b8e7-f653687da7cc-kube-api-access-6xswg\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225357 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-podres\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225357 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-sys\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225357 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225227    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-proc\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225357 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-lib-modules\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225357 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225296    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-sys\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225357 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225296    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-podres\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.225572 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.225372    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993365b5-7028-4581-b8e7-f653687da7cc-lib-modules\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.236404 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.236378    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xswg\" (UniqueName: \"kubernetes.io/projected/993365b5-7028-4581-b8e7-f653687da7cc-kube-api-access-6xswg\") pod \"perf-node-gather-daemonset-w6hzd\" (UID: \"993365b5-7028-4581-b8e7-f653687da7cc\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.326438 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.326413    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:05.455156 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.455129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"]
Jul 07 19:06:05.457357 ip-10-0-130-31 kubenswrapper[2565]: W0707 19:06:05.457332    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod993365b5_7028_4581_b8e7_f653687da7cc.slice/crio-d80ed36e3afaa01f340b9e966f14a559c25690aeab5e03930cfd659503c4cb50 WatchSource:0}: Error finding container d80ed36e3afaa01f340b9e966f14a559c25690aeab5e03930cfd659503c4cb50: Status 404 returned error can't find the container with id d80ed36e3afaa01f340b9e966f14a559c25690aeab5e03930cfd659503c4cb50
Jul 07 19:06:05.581717 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:05.581641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd" event={"ID":"993365b5-7028-4581-b8e7-f653687da7cc","Type":"ContainerStarted","Data":"d80ed36e3afaa01f340b9e966f14a559c25690aeab5e03930cfd659503c4cb50"}
Jul 07 19:06:06.126552 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:06.126524    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-dl8gt_f1ece3eb-0fdc-447a-b0db-be45265df4a3/download-server/0.log"
Jul 07 19:06:06.587907 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:06.587866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd" event={"ID":"993365b5-7028-4581-b8e7-f653687da7cc","Type":"ContainerStarted","Data":"9f36f7009582ab646a406562835e7bb76971a8f1f57725653a1adb65d1aff592"}
Jul 07 19:06:06.588098 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:06.587954    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:06.605047 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:06.604996    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd" podStartSLOduration=2.604980502 podStartE2EDuration="2.604980502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:06.602432591 +0000 UTC m=+2594.178196485" watchObservedRunningTime="2026-07-07 19:06:06.604980502 +0000 UTC m=+2594.180744429"
Jul 07 19:06:07.949998 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:07.949962    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fbx64_43eb2148-3f6d-4407-a55a-a63d49dc7324/dns/0.log"
Jul 07 19:06:07.973778 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:07.973756    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fbx64_43eb2148-3f6d-4407-a55a-a63d49dc7324/kube-rbac-proxy/0.log"
Jul 07 19:06:08.050550 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:08.050526    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v227c_2b1cab1a-2f12-4056-b400-773d0b56b4a1/dns-node-resolver/0.log"
Jul 07 19:06:08.646609 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:08.646566    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86f84896bf-jp45t_dde10661-c01b-4718-86b6-18ee7cea2ca0/registry/0.log"
Jul 07 19:06:08.705894 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:08.705867    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v67dg_2b46e1ab-8205-43d7-a162-2bd5b6838f46/node-ca/0.log"
Jul 07 19:06:09.906571 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:09.906536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-mkbdd_6c84fa22-38ff-4d6e-95f5-8b7a4fec781f/kube-auth-proxy/0.log"
Jul 07 19:06:10.091108 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:10.091078    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-z5fsx_2dfa39d3-330f-41aa-85d1-a9abb00ad516/payload-processing/0.log"
Jul 07 19:06:10.897475 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:10.897433    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rg6xl_4bce18d9-e01e-42d4-83c7-c40dfc7fc121/serve-healthcheck-canary/0.log"
Jul 07 19:06:11.483022 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:11.482995    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b885r_1faf637a-833d-4e33-8902-66cee03aa98a/kube-rbac-proxy/0.log"
Jul 07 19:06:11.506658 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:11.506629    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b885r_1faf637a-833d-4e33-8902-66cee03aa98a/exporter/0.log"
Jul 07 19:06:11.531728 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:11.531691    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b885r_1faf637a-833d-4e33-8902-66cee03aa98a/extractor/0.log"
Jul 07 19:06:12.602919 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:12.602889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-w6hzd"
Jul 07 19:06:14.225901 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:14.225870    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5c6877d755-667k6_a5f32407-ebfc-426f-b950-97b53af468c3/maas-api/0.log"
Jul 07 19:06:14.443510 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:14.443478    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787c79d564-7vvgd_cf8c1666-ca1e-454e-9ba7-1cebfc225725/manager/0.log"
Jul 07 19:06:14.727308 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:14.727275    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-w6rmf_09aefc4d-ab66-40a0-b4f4-41c5ec59e286/manager/0.log"
Jul 07 19:06:14.751651 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:14.751627    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-w6rmf_09aefc4d-ab66-40a0-b4f4-41c5ec59e286/copy-manifests/0.log"
Jul 07 19:06:15.704848 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:15.704772    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-mkbdd_6c84fa22-38ff-4d6e-95f5-8b7a4fec781f/kube-auth-proxy/0.log"
Jul 07 19:06:15.867818 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:15.867792    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-z5fsx_2dfa39d3-330f-41aa-85d1-a9abb00ad516/payload-processing/0.log"
Jul 07 19:06:16.466210 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:16.466181    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-749c476b6d-bfq9t_9bd3114e-9650-4f88-a055-18fcdff1e2e4/manager/0.log"
Jul 07 19:06:17.264555 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:17.264521    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-mkbdd_6c84fa22-38ff-4d6e-95f5-8b7a4fec781f/kube-auth-proxy/0.log"
Jul 07 19:06:17.423028 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:17.422998    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-z5fsx_2dfa39d3-330f-41aa-85d1-a9abb00ad516/payload-processing/0.log"
Jul 07 19:06:21.686380 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:21.686349    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-rfttw_18f9a32c-8f0a-42fc-9413-c8eb30e28899/migrator/0.log"
Jul 07 19:06:21.714112 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:21.714085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-rfttw_18f9a32c-8f0a-42fc-9413-c8eb30e28899/graceful-termination/0.log"
Jul 07 19:06:23.281584 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.281546    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7zx75_ac52d42f-712b-4920-bbd7-c52ca098facc/kube-multus/0.log"
Jul 07 19:06:23.390604 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.390574    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/kube-multus-additional-cni-plugins/0.log"
Jul 07 19:06:23.436655 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.436629    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/egress-router-binary-copy/0.log"
Jul 07 19:06:23.486985 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.486948    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/cni-plugins/0.log"
Jul 07 19:06:23.525757 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.525727    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/bond-cni-plugin/0.log"
Jul 07 19:06:23.554506 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.554419    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/routeoverride-cni/0.log"
Jul 07 19:06:23.584576 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.584549    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/whereabouts-cni-bincopy/0.log"
Jul 07 19:06:23.612570 ip-10-0-130-31 kubenswrapper[2565]: I0707 19:06:23.612548    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-csw9v_c8c54eb5-f3ab-4cbe-9176-f19d17dbea7e/whereabouts-cni/0.log"